2024-09-16T12:32:47.869 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T12:32:47.892 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T12:32:47.969 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: sangadi@redhat.com first_in_suite: false job_id: '7907268' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxyY0W90XibMaBvgOmsGA3hrF1PoI912Y620dP5EBwCB/QPKP38POMBenu47QIbWrikdBMiN7zQN9uouLhvyPM= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxh05cLO93mtqfeS+mekZEU0jIRv4jztc6U/HIu0gS4gTbwL2YVsDDAvw98PX6JMboE2O1LXMtwgSZyWKhIMIg= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T12:32:47.969 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T12:32:47.969 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T12:32:47.970 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T12:32:47.971 INFO:teuthology.task.internal:Checking packages... 2024-09-16T12:32:47.993 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T12:32:47.993 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T12:32:47.993 INFO:teuthology.packaging:ref: None 2024-09-16T12:32:47.993 INFO:teuthology.packaging:tag: None 2024-09-16T12:32:47.993 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:32:47.993 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:32:47.994 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:32:48.150 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-16T12:32:48.152 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T12:32:48.158 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T12:32:48.158 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T12:32:48.165 INFO:teuthology.task.internal:Saving configuration 2024-09-16T12:32:48.182 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T12:32:48.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T12:32:48.212 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:25:56.007725', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxyY0W90XibMaBvgOmsGA3hrF1PoI912Y620dP5EBwCB/QPKP38POMBenu47QIbWrikdBMiN7zQN9uouLhvyPM='} 2024-09-16T12:32:48.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:25:56.009427', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxh05cLO93mtqfeS+mekZEU0jIRv4jztc6U/HIu0gS4gTbwL2YVsDDAvw98PX6JMboE2O1LXMtwgSZyWKhIMIg='} 2024-09-16T12:32:48.233 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T12:32:48.239 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-16T12:32:48.239 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-16T12:32:48.239 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T12:32:48.293 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc085fb3ac0>, signals=[15]) 2024-09-16T12:32:48.293 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T12:32:48.299 INFO:teuthology.task.internal:Opening connections... 2024-09-16T12:32:48.300 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:32:48.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:32:48.375 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:32:48.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:32:48.447 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T12:32:48.454 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-09-16T12:32:48.474 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-09-16T12:32:48.474 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-09-16T12:32:48.531 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T12:32:48.532 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T12:32:48.533 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T12:32:48.533 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-09-16T12:32:48.549 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-09-16T12:32:48.567 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-09-16T12:32:48.567 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-09-16T12:32:48.624 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-09-16T12:32:48.624 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-09-16T12:32:48.624 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T12:32:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T12:32:48.626 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-09-16T12:32:48.640 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T12:32:48.649 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T12:32:48.656 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T12:32:48.656 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-09-16T12:32:48.658 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-09-16T12:32:48.681 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T12:32:48.686 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T12:32:48.686 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-09-16T12:32:48.716 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-09-16T12:32:48.822 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T12:32:48.869 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T12:32:48.883 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T12:32:48.883 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T12:32:48.884 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T12:32:48.884 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T12:32:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:32:48.900 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-16T12:32:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:32:48.901 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-16T12:32:48.958 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-503.el9.x86_64 2024-09-16T12:32:48.959 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-503.el9.x86_64 2024-09-16T12:32:48.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-09-16T12:32:48.960 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-503.el9.x86_64 2024-09-16T12:32:48.960 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-503.el9.x86_64 2024-09-16T12:32:48.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-09-16T12:32:52.449 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-16T12:32:53.379 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.2 MB 00:02 2024-09-16T12:33:01.290 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-16T12:33:02.412 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-16T12:33:05.833 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.5 MB 00:00 2024-09-16T12:33:07.325 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-16T12:33:07.963 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-16T12:33:09.765 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-16T12:33:10.753 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-16T12:33:14.010 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T12:33:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T12:33:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T12:33:16.110 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:33:16.119 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:33:16.120 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:33:16.121 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-09-16T12:33:16.121 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:33:16.121 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 79 M 2024-09-16T12:33:16.121 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 126 M 2024-09-16T12:33:16.121 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:33:16.350 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-16T12:33:16.703 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 75 kB/s | 23 kB 00:00 2024-09-16T12:33:18.324 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T12:33:18.324 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T12:33:18.410 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:33:18.422 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-16T12:33:18.423 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:33:18.423 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 79 M 2024-09-16T12:33:18.423 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 126 M 2024-09-16T12:33:18.423 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:33:19.002 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 43 kB/s | 23 kB 00:00 2024-09-16T12:33:19.894 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T12:33:20.636 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-16T12:33:21.378 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-16T12:33:21.378 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:33:21.379 INFO:teuthology.orchestra.run.smithi191.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-16T12:33:22.063 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:33:22.404 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:33:22.405 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:33:23.354 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:33:23.355 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:33:24.160 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.0 MB/s | 16 MB 00:07 2024-09-16T12:33:24.477 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.3 MB/s | 35 MB 00:08 2024-09-16T12:33:25.795 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:33:26.452 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T12:33:26.544 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:33:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:33:29.474 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:33:30.010 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-16T12:33:30.010 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:33:30.010 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.7 MB/s | 79 MB 00:13 2024-09-16T12:33:30.688 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:33:31.047 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:33:31.047 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:33:32.014 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:33:32.014 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:33:34.284 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:33:34.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:33:34.478 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:33:35.134 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T12:33:35.219 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:33:38.011 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:33:38.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:33:39.234 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:33:42.954 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:33:43.047 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:33:47.884 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:17.369 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:17.419 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:17.712 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:17.712 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T12:34:17.712 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:34:17.712 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T12:34:18.541 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T12:34:18.542 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:34:18.542 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:34:18.890 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T12:34:18.980 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:18.980 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T12:34:18.980 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T12:34:18.980 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T12:34:18.980 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T12:34:18.981 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-16T12:34:18.981 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T12:34:18.981 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T12:34:19.040 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:19.041 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T12:34:19.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 82.el9 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-09-16T12:34:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8274670 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-09-16T12:34:19.076 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:34:19.077 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T12:34:19.077 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T12:34:19.078 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-505.el9.x86_64 2024-09-16T12:34:19.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T12:34:19.748 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-09-16T12:34:21.076 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T12:34:21.107 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-09-16T12:34:21.114 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-16T12:34:21.153 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.c64lx7bgHB 2024-09-16T12:34:21.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c64lx7bgHB 2024-09-16T12:34:21.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.c64lx7bgHB 2024-09-16T12:34:21.362 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.c64lx7bgHB is 6KB 2024-09-16T12:34:21.382 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.c64lx7bgHB 2024-09-16T12:34:21.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T12:34:21.459 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T12:34:21.459 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T12:34:21.459 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T12:34:21.459 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T12:34:21.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T12:34:21.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-09-16T12:34:26.332 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:26.749 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:26.749 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T12:34:26.749 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T12:34:26.749 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:34:27.608 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:34:27.938 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T12:34:28.027 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:28.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T12:34:28.027 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T12:34:28.027 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T12:34:28.027 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T12:34:28.027 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-16T12:34:28.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T12:34:28.028 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T12:34:28.087 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T12:34:28.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T12:34:28.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-09-16T12:34:28.127 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-09-16T12:34:28.127 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 82.el9 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8274670 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-09-16T12:34:28.128 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:34:28.129 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T12:34:28.130 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T12:34:28.130 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-505.el9.x86_64 2024-09-16T12:34:28.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T12:34:28.729 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-09-16T12:34:30.047 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T12:34:30.086 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-09-16T12:34:30.089 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-16T12:34:30.104 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.xXzifebl1q 2024-09-16T12:34:30.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xXzifebl1q 2024-09-16T12:34:30.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.xXzifebl1q 2024-09-16T12:34:30.330 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.xXzifebl1q is 6KB 2024-09-16T12:34:30.378 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.xXzifebl1q 2024-09-16T12:34:30.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T12:34:30.455 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T12:34:30.456 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T12:34:30.456 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T12:34:30.456 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T12:34:30.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T12:34:30.616 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-09-16T12:34:51.629 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-16T12:34:51.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-16T12:34:51.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:35:00.627 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-16T12:35:00.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T12:35:00.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:35:51.684 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T12:36:00.639 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T12:36:00.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-16T12:36:00.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:36:01.011 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-16T12:36:01.385 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-16T12:36:01.385 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T12:36:01.385 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-16T12:36:01.400 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-505.el9.x86_64 2024-09-16T12:36:01.400 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T12:36:01.400 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T12:36:01.400 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T12:36:02.401 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-16T12:36:02.402 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T12:36:02.504 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-09-16T12:36:02.545 DEBUG:teuthology.parallel:result is None 2024-09-16T12:36:09.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T12:36:09.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:36:09.905 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-16T12:36:10.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T12:36:10.342 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T12:36:10.342 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-16T12:36:10.358 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-505.el9.x86_64 2024-09-16T12:36:10.358 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T12:36:10.358 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T12:36:10.358 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T12:36:11.360 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-16T12:36:11.360 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T12:36:11.460 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-09-16T12:36:11.486 DEBUG:teuthology.parallel:result is None 2024-09-16T12:36:11.486 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T12:36:11.496 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T12:36:11.496 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T12:36:11.498 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T12:36:11.518 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T12:36:11.524 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T12:36:11.532 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T12:36:11.532 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T12:36:11.552 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T12:36:11.593 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T12:36:11.599 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T12:36:11.599 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T12:36:11.613 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T12:36:11.644 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:36:11.654 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:36:11.657 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:36:11.668 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:36:11.670 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T12:36:11.677 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T12:36:11.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T12:36:11.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T12:36:11.737 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T12:36:11.749 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T12:36:11.749 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T12:36:11.764 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T12:36:11.793 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:36:11.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:36:11.899 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:36:11.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:36:12.024 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:36:12.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T12:36:12.090 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:36:12.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:36:12.186 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:36:12.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:36:12.311 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:36:12.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T12:36:12.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-09-16T12:36:12.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-09-16T12:36:12.425 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T12:36:12.475 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T12:36:12.886 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T12:36:12.894 INFO:teuthology.task.internal:Starting timer... 2024-09-16T12:36:12.894 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T12:36:12.905 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T12:36:12.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-09-16T12:36:12.959 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-09-16T12:36:12.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-09-16T12:36:13.031 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-09-16T12:36:13.033 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T12:36:13.033 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-09-16T12:36:13.053 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-09-16T12:36:13.054 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-09-16T12:36:13.090 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-09-16T12:36:13.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi177.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-09-16T12:36:13.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T12:36:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:36:13.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T12:36:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:36:13.156 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T12:36:13.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-09-16T12:36:13.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-09-16T12:36:13.227 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T12:36:13.237 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T12:36:13.240 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T12:36:13.354 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T12:36:13.364 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T12:36:13.364 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi177.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T12:39:54.469 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi177.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-09-16T12:39:54.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T12:39:54.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:39:54.547 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-16T12:39:54.624 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T12:39:54.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-16T12:39:54.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:39:54.697 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-16T12:39:54.780 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-16T12:39:54.781 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T12:39:54.791 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T12:39:54.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T12:39:54.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T12:39:54.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T12:39:54.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T12:39:54.823 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T12:39:54.837 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T12:39:54.851 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T12:39:54.862 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-09-16T12:39:54.865 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T12:39:54.873 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-09-16T12:39:54.887 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T12:39:54.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-09-16T12:39:54.900 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-09-16T12:39:54.901 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T12:39:54.914 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T12:39:54.928 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T12:39:54.958 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-09-16T12:39:54.961 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T12:39:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-09-16T12:39:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.962 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.985 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.988 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T12:39:54.989 INFO:teuthology.run_tasks:Running task install... 2024-09-16T12:39:55.003 DEBUG:teuthology.task.install:project ceph 2024-09-16T12:39:55.003 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T12:39:55.003 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-16T12:39:55.003 INFO:teuthology.task.install:Using flavor: default 2024-09-16T12:39:55.009 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T12:39:55.009 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-16T12:39:55.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T12:39:55.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T12:39:55.574 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:45 ago on Mon 16 Sep 2024 12:39:10 PM UTC. 2024-09-16T12:39:55.584 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:45 ago on Mon 16 Sep 2024 12:39:10 PM UTC. 2024-09-16T12:39:55.662 INFO:teuthology.orchestra.run.smithi177.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T12:39:55.671 INFO:teuthology.orchestra.run.smithi191.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T12:39:55.705 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:39:55.706 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:39:55.706 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:39:55.712 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:39:55.714 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:39:55.714 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:39:55.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo dnf -y copr enable ceph/el9 2024-09-16T12:39:55.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo dnf -y copr enable ceph/el9 2024-09-16T12:39:55.944 INFO:teuthology.orchestra.run.smithi177.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:of the main distribution, and quality may vary. 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:, 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:and packages are not held to any quality or security level. 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T12:39:55.945 INFO:teuthology.orchestra.run.smithi177.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T12:39:55.953 INFO:teuthology.orchestra.run.smithi191.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:of the main distribution, and quality may vary. 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:, 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:and packages are not held to any quality or security level. 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T12:39:55.954 INFO:teuthology.orchestra.run.smithi191.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T12:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout:Repository successfully enabled. 2024-09-16T12:39:56.058 INFO:teuthology.orchestra.run.smithi177.stdout:Repository successfully enabled. 2024-09-16T12:39:56.090 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T12:39:56.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:39:56.096 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T12:39:56.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:39:56.244 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T12:39:56.244 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T12:39:56.251 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T12:39:56.251 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T12:39:56.359 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T12:39:56.359 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:39:56.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T12:39:56.365 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T12:39:56.365 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:39:56.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T12:39:56.384 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T12:39:56.385 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T12:39:56.390 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T12:39:56.390 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T12:39:56.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T12:39:56.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T12:39:56.526 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T12:39:56.531 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T12:39:56.590 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-09-16T12:39:56.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-09-16T12:39:56.594 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-09-16T12:39:56.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-09-16T12:39:56.821 INFO:teuthology.orchestra.run.smithi177.stdout:45 files removed 2024-09-16T12:39:56.842 INFO:teuthology.orchestra.run.smithi191.stdout:45 files removed 2024-09-16T12:39:56.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-09-16T12:39:56.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-09-16T12:39:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 308 kB/s | 77 kB 00:00 2024-09-16T12:39:57.407 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 284 kB/s | 77 kB 00:00 2024-09-16T12:39:57.678 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-16T12:39:57.700 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-16T12:39:57.929 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 8.3 kB/s | 1.8 kB 00:00 2024-09-16T12:39:57.984 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 7.2 kB/s | 1.8 kB 00:00 2024-09-16T12:39:58.121 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for el9 owned by ceph 48 kB/s | 7.6 kB 00:00 2024-09-16T12:39:58.209 INFO:teuthology.orchestra.run.smithi177.stdout:Copr repo for el9 owned by ceph 40 kB/s | 7.6 kB 00:00 2024-09-16T12:40:00.901 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-16T12:40:01.096 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-16T12:40:02.586 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 36 MB/s | 20 MB 00:00 2024-09-16T12:40:03.056 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-16T12:40:07.319 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.5 MB 00:00 2024-09-16T12:40:08.523 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T12:40:08.820 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 19 kB 00:00 2024-09-16T12:40:10.066 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T12:40:10.240 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-16T12:40:11.628 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T12:40:15.405 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T12:40:17.042 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T12:40:17.979 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:40:18.011 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T12:40:18.012 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout:Install 22 Packages 2024-09-16T12:40:18.013 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-09-16T12:40:18.014 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:18.014 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 58 M 2024-09-16T12:40:18.014 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:40:19.097 INFO:teuthology.orchestra.run.smithi177.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 8.9 MB/s | 5.0 MB 00:00 2024-09-16T12:40:19.147 INFO:teuthology.orchestra.run.smithi177.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 506 kB/s | 25 kB 00:00 2024-09-16T12:40:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:(3/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 19 MB/s | 14 MB 00:00 2024-09-16T12:40:19.356 INFO:teuthology.orchestra.run.smithi177.stdout:(4/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 3.0 MB/s | 637 kB 00:00 2024-09-16T12:40:19.382 INFO:teuthology.orchestra.run.smithi177.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 6.7 MB/s | 458 kB 00:00 2024-09-16T12:40:19.432 INFO:teuthology.orchestra.run.smithi177.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 908 kB/s | 45 kB 00:00 2024-09-16T12:40:19.507 INFO:teuthology.orchestra.run.smithi177.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 1.6 MB/s | 120 kB 00:00 2024-09-16T12:40:19.517 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T12:40:19.537 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T12:40:19.538 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout:Install 22 Packages 2024-09-16T12:40:19.539 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-09-16T12:40:19.540 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:19.540 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 58 M 2024-09-16T12:40:19.540 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:40:19.608 INFO:teuthology.orchestra.run.smithi177.stdout:(8/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 20 MB/s | 4.9 MB 00:00 2024-09-16T12:40:19.633 INFO:teuthology.orchestra.run.smithi177.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.2 MB/s | 156 kB 00:00 2024-09-16T12:40:19.675 INFO:teuthology.orchestra.run.smithi177.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 4.7 MB/s | 320 kB 00:00 2024-09-16T12:40:19.700 INFO:teuthology.orchestra.run.smithi177.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.3 MB/s | 295 kB 00:00 2024-09-16T12:40:19.725 INFO:teuthology.orchestra.run.smithi177.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-16T12:40:19.976 INFO:teuthology.orchestra.run.smithi177.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 16 MB/s | 22 MB 00:01 2024-09-16T12:40:20.260 INFO:teuthology.orchestra.run.smithi177.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 85 kB/s | 45 kB 00:00 2024-09-16T12:40:20.360 INFO:teuthology.orchestra.run.smithi191.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 9.5 MB/s | 5.0 MB 00:00 2024-09-16T12:40:20.418 INFO:teuthology.orchestra.run.smithi191.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 433 kB/s | 25 kB 00:00 2024-09-16T12:40:20.476 INFO:teuthology.orchestra.run.smithi177.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 206 kB/s | 160 kB 00:00 2024-09-16T12:40:20.501 INFO:teuthology.orchestra.run.smithi191.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-16T12:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 6.0 MB/s | 458 kB 00:00 2024-09-16T12:40:20.584 INFO:teuthology.orchestra.run.smithi177.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-16T12:40:20.635 INFO:teuthology.orchestra.run.smithi177.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 779 kB/s | 292 kB 00:00 2024-09-16T12:40:20.660 INFO:teuthology.orchestra.run.smithi177.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-16T12:40:20.694 INFO:teuthology.orchestra.run.smithi177.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.5 MB/s | 67 kB 00:00 2024-09-16T12:40:20.728 INFO:teuthology.orchestra.run.smithi177.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-09-16T12:40:20.786 INFO:teuthology.orchestra.run.smithi177.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 818 kB/s | 662 kB 00:00 2024-09-16T12:40:20.828 INFO:teuthology.orchestra.run.smithi177.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T12:40:20.877 INFO:teuthology.orchestra.run.smithi191.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-16T12:40:21.028 INFO:teuthology.orchestra.run.smithi191.stdout:(6/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 12 MB/s | 14 MB 00:01 2024-09-16T12:40:21.036 INFO:teuthology.orchestra.run.smithi177.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 10 MB/s | 3.1 MB 00:00 2024-09-16T12:40:21.044 INFO:teuthology.orchestra.run.smithi191.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 271 kB/s | 45 kB 00:00 2024-09-16T12:40:21.078 INFO:teuthology.orchestra.run.smithi191.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.4 MB/s | 120 kB 00:00 2024-09-16T12:40:21.086 INFO:teuthology.orchestra.run.smithi177.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 9.8 MB/s | 2.9 MB 00:00 2024-09-16T12:40:21.091 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:21.091 INFO:teuthology.orchestra.run.smithi177.stdout:Total 19 MB/s | 58 MB 00:03 2024-09-16T12:40:21.103 INFO:teuthology.orchestra.run.smithi191.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-16T12:40:21.128 INFO:teuthology.orchestra.run.smithi191.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-16T12:40:21.154 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:40:21.162 INFO:teuthology.orchestra.run.smithi191.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-16T12:40:21.187 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:40:21.187 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:40:21.188 INFO:teuthology.orchestra.run.smithi191.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.7 MB/s | 100 kB 00:00 2024-09-16T12:40:21.396 INFO:teuthology.orchestra.run.smithi191.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 14 MB/s | 22 MB 00:01 2024-09-16T12:40:21.505 INFO:teuthology.orchestra.run.smithi191.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 143 kB/s | 45 kB 00:00 2024-09-16T12:40:21.547 INFO:teuthology.orchestra.run.smithi191.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-16T12:40:21.622 INFO:teuthology.orchestra.run.smithi191.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-16T12:40:21.647 INFO:teuthology.orchestra.run.smithi191.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-09-16T12:40:21.672 INFO:teuthology.orchestra.run.smithi191.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-16T12:40:21.731 INFO:teuthology.orchestra.run.smithi191.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-09-16T12:40:21.756 INFO:teuthology.orchestra.run.smithi191.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T12:40:21.781 INFO:teuthology.orchestra.run.smithi191.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 716 kB/s | 67 kB 00:00 2024-09-16T12:40:21.851 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:40:21.851 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:40:21.907 INFO:teuthology.orchestra.run.smithi191.stdout:(22/24): librados2-17.2.7-1643.gd75f2382.el9.x8 21 MB/s | 3.1 MB 00:00 2024-09-16T12:40:21.949 INFO:teuthology.orchestra.run.smithi191.stdout:(23/24): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-09-16T12:40:22.016 INFO:teuthology.orchestra.run.smithi191.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-16T12:40:22.021 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:22.021 INFO:teuthology.orchestra.run.smithi191.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-16T12:40:22.085 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:40:22.118 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:40:22.118 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:40:22.769 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:40:22.769 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:40:23.073 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:40:23.407 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T12:40:23.513 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:23.605 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:23.683 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T12:40:23.706 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:23.796 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:23.842 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T12:40:23.871 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T12:40:23.993 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:40:24.307 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T12:40:24.336 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T12:40:24.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T12:40:24.390 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T12:40:24.419 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T12:40:24.426 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:24.461 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T12:40:24.499 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T12:40:24.519 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:24.561 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T12:40:24.589 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T12:40:24.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T12:40:24.654 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T12:40:24.661 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:24.700 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T12:40:24.738 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T12:40:24.749 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:24.771 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T12:40:24.794 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T12:40:24.833 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T12:40:25.062 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T12:40:25.109 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T12:40:25.200 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T12:40:25.272 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T12:40:25.278 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T12:40:25.303 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T12:40:25.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T12:40:25.351 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T12:40:25.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T12:40:25.423 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T12:40:25.460 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T12:40:25.522 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T12:40:25.550 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T12:40:25.599 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T12:40:25.653 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T12:40:25.692 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T12:40:25.725 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T12:40:26.016 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T12:40:26.062 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T12:40:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T12:40:26.226 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T12:40:26.256 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T12:40:27.733 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:27.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:28.298 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T12:40:28.425 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T12:40:28.425 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T12:40:28.425 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T12:40:28.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:28.454 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:28.665 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:28.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:29.237 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T12:40:29.272 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T12:40:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T12:40:29.365 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T12:40:29.365 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T12:40:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:29.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-09-16T12:40:38.155 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-09-16T12:40:39.362 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:40:39.566 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:39.568 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T12:40:39.657 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T12:40:39.657 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T12:40:40.769 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T12:40:40.769 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:40:40.769 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T12:40:40.769 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:40:40.769 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:40:40.770 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:40.771 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T12:40:40.861 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T12:40:40.861 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:40.888 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:40.888 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T12:40:40.889 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T12:40:40.890 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T12:40:41.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:41.722 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:40:42.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-09-16T12:40:42.159 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:42.159 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T12:40:42.159 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T12:40:42.160 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T12:40:42.161 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T12:40:42.617 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:40:42.748 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:40:42.748 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-09-16T12:40:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:42.752 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 48 M 2024-09-16T12:40:42.752 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 193 M 2024-09-16T12:40:42.753 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:40:43.020 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 664 kB/s | 163 kB 00:00 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.084 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.085 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:43.086 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:40:43.170 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 769 kB/s | 304 kB 00:00 2024-09-16T12:40:43.220 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-09-16T12:40:43.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-09-16T12:40:44.015 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:40:44.151 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:40:44.151 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-16T12:40:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:44.153 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 48 M 2024-09-16T12:40:44.153 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 193 M 2024-09-16T12:40:44.153 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:40:44.924 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 331 kB/s | 163 kB 00:00 2024-09-16T12:40:45.207 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 392 kB/s | 304 kB 00:00 2024-09-16T12:40:45.533 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 105 kB/s | 64 kB 00:00 2024-09-16T12:40:45.753 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 16 MB/s | 48 MB 00:02 2024-09-16T12:40:45.755 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:45.756 INFO:teuthology.orchestra.run.smithi177.stdout:Total 16 MB/s | 48 MB 00:03 2024-09-16T12:40:45.783 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:40:45.803 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:40:45.803 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:40:46.171 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:40:46.171 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:40:46.733 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:40:46.813 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T12:40:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T12:40:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T12:40:47.033 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 18 MB/s | 48 MB 00:02 2024-09-16T12:40:47.035 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:47.035 INFO:teuthology.orchestra.run.smithi191.stdout:Total 17 MB/s | 48 MB 00:02 2024-09-16T12:40:47.063 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:40:47.082 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:40:47.082 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:40:47.448 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:40:47.449 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:40:47.967 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:40:48.049 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T12:40:48.092 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T12:40:48.118 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T12:40:51.304 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T12:40:51.378 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T12:40:52.561 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T12:40:52.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T12:40:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T12:40:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T12:40:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T12:40:53.602 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T12:40:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T12:40:53.922 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:53.923 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:40:54.252 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-09-16T12:40:54.825 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:40:54.968 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:40:54.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:54.970 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:40:54.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:54.970 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:40:54.970 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T12:40:54.971 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T12:40:54.972 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:40:54.973 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:40:54.974 INFO:teuthology.orchestra.run.smithi177.stdout:Install 34 Packages 2024-09-16T12:40:54.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:40:54.974 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 27 M 2024-09-16T12:40:54.974 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 96 M 2024-09-16T12:40:54.974 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:40:55.020 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T12:40:55.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T12:40:55.021 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T12:40:55.021 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T12:40:55.328 INFO:teuthology.orchestra.run.smithi177.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 56 kB/s | 6.5 kB 00:00 2024-09-16T12:40:55.354 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T12:40:55.354 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:55.355 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:40:55.503 INFO:teuthology.orchestra.run.smithi177.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.8 MB/s | 1.4 MB 00:00 2024-09-16T12:40:55.553 INFO:teuthology.orchestra.run.smithi177.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.1 MB/s | 2.1 MB 00:00 2024-09-16T12:40:55.620 INFO:teuthology.orchestra.run.smithi177.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 13 MB/s | 3.9 MB 00:00 2024-09-16T12:40:55.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-09-16T12:40:55.646 INFO:teuthology.orchestra.run.smithi177.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-16T12:40:55.805 INFO:teuthology.orchestra.run.smithi177.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T12:40:55.855 INFO:teuthology.orchestra.run.smithi177.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-16T12:40:56.005 INFO:teuthology.orchestra.run.smithi177.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-16T12:40:56.030 INFO:teuthology.orchestra.run.smithi177.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 772 kB/s | 135 kB 00:00 2024-09-16T12:40:56.063 INFO:teuthology.orchestra.run.smithi177.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-16T12:40:56.089 INFO:teuthology.orchestra.run.smithi177.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T12:40:56.147 INFO:teuthology.orchestra.run.smithi177.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 878 kB/s | 125 kB 00:00 2024-09-16T12:40:56.198 INFO:teuthology.orchestra.run.smithi177.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T12:40:56.213 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:40:56.223 INFO:teuthology.orchestra.run.smithi177.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T12:40:56.273 INFO:teuthology.orchestra.run.smithi177.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 613 kB/s | 77 kB 00:00 2024-09-16T12:40:56.299 INFO:teuthology.orchestra.run.smithi177.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 505 kB/s | 51 kB 00:00 2024-09-16T12:40:56.324 INFO:teuthology.orchestra.run.smithi177.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-09-16T12:40:56.355 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:40:56.357 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:56.357 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T12:40:56.358 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T12:40:56.359 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:40:56.360 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:40:56.361 INFO:teuthology.orchestra.run.smithi191.stdout:Install 34 Packages 2024-09-16T12:40:56.361 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:40:56.361 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 27 M 2024-09-16T12:40:56.361 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 96 M 2024-09-16T12:40:56.361 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:40:56.374 INFO:teuthology.orchestra.run.smithi177.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-16T12:40:56.408 INFO:teuthology.orchestra.run.smithi177.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 341 kB/s | 11 kB 00:00 2024-09-16T12:40:56.433 INFO:teuthology.orchestra.run.smithi177.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T12:40:56.458 INFO:teuthology.orchestra.run.smithi177.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-16T12:40:56.475 INFO:teuthology.orchestra.run.smithi177.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 552 kB/s | 23 kB 00:00 2024-09-16T12:40:56.493 INFO:teuthology.orchestra.run.smithi177.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-16T12:40:56.509 INFO:teuthology.orchestra.run.smithi177.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 584 kB/s | 20 kB 00:00 2024-09-16T12:40:56.534 INFO:teuthology.orchestra.run.smithi177.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-16T12:40:56.581 INFO:teuthology.orchestra.run.smithi177.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-16T12:40:56.610 INFO:teuthology.orchestra.run.smithi177.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-16T12:40:56.627 INFO:teuthology.orchestra.run.smithi177.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 364 kB/s | 16 kB 00:00 2024-09-16T12:40:56.652 INFO:teuthology.orchestra.run.smithi177.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T12:40:56.669 INFO:teuthology.orchestra.run.smithi177.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-16T12:40:56.703 INFO:teuthology.orchestra.run.smithi177.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-16T12:40:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-16T12:40:56.745 INFO:teuthology.orchestra.run.smithi177.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-09-16T12:40:56.778 INFO:teuthology.orchestra.run.smithi177.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T12:40:56.782 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:56.782 INFO:teuthology.orchestra.run.smithi177.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-16T12:40:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-16T12:40:56.907 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:40:56.944 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:40:56.944 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:40:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.7 MB/s | 1.4 MB 00:00 2024-09-16T12:40:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-16T12:40:57.217 INFO:teuthology.orchestra.run.smithi191.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-16T12:40:57.242 INFO:teuthology.orchestra.run.smithi191.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.3 MB/s | 240 kB 00:00 2024-09-16T12:40:57.342 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:40:57.343 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:40:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T12:40:57.635 INFO:teuthology.orchestra.run.smithi191.stdout:(7/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 27 MB/s | 15 MB 00:00 2024-09-16T12:40:57.660 INFO:teuthology.orchestra.run.smithi191.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 441 kB/s | 106 kB 00:00 2024-09-16T12:40:57.710 INFO:teuthology.orchestra.run.smithi191.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T12:40:57.769 INFO:teuthology.orchestra.run.smithi191.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T12:40:57.827 INFO:teuthology.orchestra.run.smithi191.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.1 MB/s | 1.2 MB 00:00 2024-09-16T12:40:57.886 INFO:teuthology.orchestra.run.smithi191.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 539 kB/s | 135 kB 00:00 2024-09-16T12:40:57.928 INFO:teuthology.orchestra.run.smithi191.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T12:40:57.953 INFO:teuthology.orchestra.run.smithi191.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-16T12:40:58.004 INFO:teuthology.orchestra.run.smithi191.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 659 kB/s | 77 kB 00:00 2024-09-16T12:40:58.029 INFO:teuthology.orchestra.run.smithi191.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-16T12:40:58.054 INFO:teuthology.orchestra.run.smithi191.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-09-16T12:40:58.104 INFO:teuthology.orchestra.run.smithi191.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-16T12:40:58.129 INFO:teuthology.orchestra.run.smithi191.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 463 kB/s | 11 kB 00:00 2024-09-16T12:40:58.155 INFO:teuthology.orchestra.run.smithi191.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T12:40:58.180 INFO:teuthology.orchestra.run.smithi191.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 349 kB/s | 18 kB 00:00 2024-09-16T12:40:58.206 INFO:teuthology.orchestra.run.smithi191.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-16T12:40:58.231 INFO:teuthology.orchestra.run.smithi191.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T12:40:58.256 INFO:teuthology.orchestra.run.smithi191.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T12:40:58.281 INFO:teuthology.orchestra.run.smithi191.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-16T12:40:58.307 INFO:teuthology.orchestra.run.smithi191.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T12:40:58.332 INFO:teuthology.orchestra.run.smithi191.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-16T12:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T12:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T12:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-16T12:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T12:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T12:40:58.492 INFO:teuthology.orchestra.run.smithi191.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-09-16T12:40:58.534 INFO:teuthology.orchestra.run.smithi191.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-16T12:40:58.538 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:40:58.539 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-16T12:40:58.587 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:40:58.656 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T12:40:58.672 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:40:58.688 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T12:40:58.709 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:40:58.709 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:40:58.721 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T12:40:58.812 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T12:40:58.908 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T12:40:58.949 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T12:40:58.996 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T12:40:59.025 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T12:40:59.060 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T12:40:59.094 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T12:40:59.104 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:40:59.105 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:40:59.134 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T12:40:59.169 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T12:40:59.204 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T12:40:59.252 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T12:40:59.284 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T12:40:59.356 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T12:40:59.493 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T12:40:59.582 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T12:40:59.636 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T12:40:59.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T12:40:59.738 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T12:40:59.804 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T12:40:59.988 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T12:41:00.029 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T12:41:00.106 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T12:41:00.194 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T12:41:00.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T12:41:00.350 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T12:41:00.361 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:00.453 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T12:41:00.485 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T12:41:00.500 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T12:41:00.522 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T12:41:00.539 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T12:41:00.565 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T12:41:00.566 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:00.566 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T12:41:00.566 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T12:41:00.566 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T12:41:00.566 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:00.622 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T12:41:00.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T12:41:00.749 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T12:41:00.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T12:41:00.823 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T12:41:00.857 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T12:41:00.890 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T12:41:00.937 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T12:41:00.975 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T12:41:01.017 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T12:41:01.073 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T12:41:01.116 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T12:41:01.189 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T12:41:01.328 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T12:41:01.418 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T12:41:01.477 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T12:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T12:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T12:41:01.619 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T12:41:01.813 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T12:41:01.861 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T12:41:01.873 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T12:41:01.900 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T12:41:01.900 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:01.900 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T12:41:01.901 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T12:41:01.901 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T12:41:01.901 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:01.930 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T12:41:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T12:41:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T12:41:02.192 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T12:41:02.274 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T12:41:02.298 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:02.341 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T12:41:02.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T12:41:02.407 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:02.499 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T12:41:02.523 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:02.587 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T12:41:03.731 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T12:41:03.758 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T12:41:03.759 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:03.759 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T12:41:03.759 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T12:41:03.759 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T12:41:03.759 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:04.132 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T12:41:04.156 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T12:41:04.241 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T12:41:04.242 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T12:41:04.243 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T12:41:04.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T12:41:04.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T12:41:04.389 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T12:41:04.390 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:04.390 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T12:41:04.390 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T12:41:04.390 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T12:41:04.390 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:04.454 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T12:41:05.176 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T12:41:05.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:05.176 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:05.176 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:05.176 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T12:41:05.177 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T12:41:05.178 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:05.179 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:05.535 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-09-16T12:41:06.110 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:06.198 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T12:41:06.198 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T12:41:06.198 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T12:41:06.198 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T12:41:06.198 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T12:41:06.199 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T12:41:06.200 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T12:41:06.201 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T12:41:06.201 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:06.241 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:06.242 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:06.242 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:06.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-09-16T12:41:06.894 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:07.027 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:07.028 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:07.029 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:07.029 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 83 k 2024-09-16T12:41:07.029 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 375 k 2024-09-16T12:41:07.029 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T12:41:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:07.186 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:07.222 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-16T12:41:07.223 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:07.223 INFO:teuthology.orchestra.run.smithi177.stdout:Total 424 kB/s | 83 kB 00:00 2024-09-16T12:41:07.223 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:07.228 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:07.228 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:07.230 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:07.231 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:07.300 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:07.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-09-16T12:41:07.796 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:07.876 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:08.133 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:08.221 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:08.222 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:08.264 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:08.265 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:08.265 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:08.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-09-16T12:41:08.456 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:08.456 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:08.457 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:08.457 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:08.457 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:08.457 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:08.710 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T12:41:08.911 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:09.043 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 83 k 2024-09-16T12:41:09.044 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 375 k 2024-09-16T12:41:09.045 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:09.250 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 406 kB/s | 83 kB 00:00 2024-09-16T12:41:09.250 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:09.250 INFO:teuthology.orchestra.run.smithi191.stdout:Total 403 kB/s | 83 kB 00:00 2024-09-16T12:41:09.251 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:09.254 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:09.254 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:09.256 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:09.256 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:09.288 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:09.334 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:09.423 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:09.424 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:09.425 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:09.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:09.425 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 137 k 2024-09-16T12:41:09.425 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 379 k 2024-09-16T12:41:09.425 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:09.636 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 651 kB/s | 137 kB 00:00 2024-09-16T12:41:09.637 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:09.637 INFO:teuthology.orchestra.run.smithi177.stdout:Total 646 kB/s | 137 kB 00:00 2024-09-16T12:41:09.637 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:09.645 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:09.645 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:09.702 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:09.702 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:09.747 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:09.847 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:09.860 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:09.912 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:09.937 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:09.937 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:09.937 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T12:41:09.937 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:10.193 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:10.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:10.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:10.454 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:10.454 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:10.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:10.454 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:10.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T12:41:10.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:10.836 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:10.836 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:10.836 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:10.836 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:10.836 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:11.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-09-16T12:41:11.268 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:11.402 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:11.402 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:11.402 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-16T12:41:11.402 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:11.403 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 137 k 2024-09-16T12:41:11.404 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 379 k 2024-09-16T12:41:11.404 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:11.587 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 750 kB/s | 137 kB 00:00 2024-09-16T12:41:11.588 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:11.588 INFO:teuthology.orchestra.run.smithi191.stdout:Total 741 kB/s | 137 kB 00:00 2024-09-16T12:41:11.589 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:11.596 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:11.596 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:11.657 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:11.657 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:11.675 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:11.769 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:11.811 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:11.812 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:11.812 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:11.889 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:11.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T12:41:11.913 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:11.913 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:11.914 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T12:41:11.914 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:12.466 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:12.599 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T12:41:12.600 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout:Install 7 Packages 2024-09-16T12:41:12.601 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:12.602 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.7 M 2024-09-16T12:41:12.602 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 80 M 2024-09-16T12:41:12.602 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:12.867 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:12.914 INFO:teuthology.orchestra.run.smithi177.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 167 kB/s | 24 kB 00:00 2024-09-16T12:41:12.940 INFO:teuthology.orchestra.run.smithi177.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 85 kB/s | 15 kB 00:00 2024-09-16T12:41:12.982 INFO:teuthology.orchestra.run.smithi177.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-16T12:41:13.007 INFO:teuthology.orchestra.run.smithi177.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T12:41:13.041 INFO:teuthology.orchestra.run.smithi177.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 405 kB/s | 41 kB 00:00 2024-09-16T12:41:13.075 INFO:teuthology.orchestra.run.smithi177.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-09-16T12:41:13.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-09-16T12:41:13.175 INFO:teuthology.orchestra.run.smithi177.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.3 MB/s | 3.4 MB 00:00 2024-09-16T12:41:13.178 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:13.179 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.3 MB/s | 3.7 MB 00:00 2024-09-16T12:41:13.179 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:13.187 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:13.187 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:13.269 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:13.270 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:13.581 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:13.634 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T12:41:13.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T12:41:13.671 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:13.723 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T12:41:13.755 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T12:41:13.765 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:13.792 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T12:41:13.808 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:13.809 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:13.810 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:13.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T12:41:14.445 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:14.577 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:14.578 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:14.578 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:14.578 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:14.579 INFO:teuthology.orchestra.run.smithi191.stdout:Install 7 Packages 2024-09-16T12:41:14.580 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:14.580 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.7 M 2024-09-16T12:41:14.580 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 80 M 2024-09-16T12:41:14.580 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:14.793 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T12:41:14.837 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T12:41:14.891 INFO:teuthology.orchestra.run.smithi191.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 155 kB/s | 24 kB 00:00 2024-09-16T12:41:14.916 INFO:teuthology.orchestra.run.smithi191.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 80 kB/s | 15 kB 00:00 2024-09-16T12:41:14.959 INFO:teuthology.orchestra.run.smithi191.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-16T12:41:14.984 INFO:teuthology.orchestra.run.smithi191.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T12:41:15.009 INFO:teuthology.orchestra.run.smithi191.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-16T12:41:15.042 INFO:teuthology.orchestra.run.smithi191.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-16T12:41:15.143 INFO:teuthology.orchestra.run.smithi191.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.2 MB/s | 3.4 MB 00:00 2024-09-16T12:41:15.146 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:15.146 INFO:teuthology.orchestra.run.smithi191.stdout:Total 6.4 MB/s | 3.7 MB 00:00 2024-09-16T12:41:15.146 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:15.154 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:15.155 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:15.232 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:15.236 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:15.562 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:15.627 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T12:41:15.661 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T12:41:15.716 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T12:41:15.756 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T12:41:15.785 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T12:41:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T12:41:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T12:41:15.816 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T12:41:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T12:41:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T12:41:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T12:41:15.817 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:16.084 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T12:41:16.085 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T12:41:16.085 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T12:41:16.085 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T12:41:16.085 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:16.085 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:16.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T12:41:16.772 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T12:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T12:41:16.900 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:17.033 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:17.034 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-16T12:41:17.034 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:17.034 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T12:41:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:17.036 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 38 M 2024-09-16T12:41:17.037 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 212 M 2024-09-16T12:41:17.037 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:17.565 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 934 kB/s | 191 kB 00:00 2024-09-16T12:41:17.666 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-16T12:41:17.691 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-16T12:41:17.808 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-16T12:41:17.824 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 136 kB/s | 18 kB 00:00 2024-09-16T12:41:17.866 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T12:41:17.866 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T12:41:17.866 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T12:41:17.866 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T12:41:17.867 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T12:41:17.867 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T12:41:17.867 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T12:41:17.900 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 419 kB/s | 38 kB 00:00 2024-09-16T12:41:18.025 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-16T12:41:18.109 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-16T12:41:18.135 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:18.136 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:18.159 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-09-16T12:41:18.243 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 5.2 MB/s | 3.0 MB 00:00 2024-09-16T12:41:18.318 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-16T12:41:18.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T12:41:18.585 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-09-16T12:41:18.586 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:18.586 INFO:teuthology.orchestra.run.smithi177.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-16T12:41:18.936 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:02 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:18.938 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:18.964 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:18.964 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:19.069 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T12:41:19.070 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-09-16T12:41:19.071 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:19.072 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 38 M 2024-09-16T12:41:19.072 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 212 M 2024-09-16T12:41:19.072 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:19.454 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:19.455 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:19.459 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-16T12:41:19.526 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-16T12:41:19.559 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 611 kB/s | 191 kB 00:00 2024-09-16T12:41:19.676 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-16T12:41:19.701 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-16T12:41:19.727 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 769 kB/s | 38 kB 00:00 2024-09-16T12:41:19.836 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-09-16T12:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-16T12:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-09-16T12:41:19.995 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-16T12:41:20.230 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:20.315 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T12:41:20.320 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-09-16T12:41:20.367 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T12:41:20.395 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.1 MB/s | 4.9 MB 00:00 2024-09-16T12:41:20.396 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:20.396 INFO:teuthology.orchestra.run.smithi191.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-16T12:41:20.464 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T12:41:20.509 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T12:41:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T12:41:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T12:41:20.752 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:20.778 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:20.778 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T12:41:21.267 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:21.268 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:21.447 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T12:41:21.578 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T12:41:22.033 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:22.103 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T12:41:22.160 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T12:41:22.259 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T12:41:22.294 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T12:41:22.480 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T12:41:22.527 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T12:41:22.604 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T12:41:22.823 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T12:41:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T12:41:23.389 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T12:41:23.876 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T12:41:23.908 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T12:41:24.633 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T12:41:25.674 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T12:41:25.701 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T12:41:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T12:41:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T12:41:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T12:41:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T12:41:26.894 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:26.895 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:27.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-09-16T12:41:27.855 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:27.996 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T12:41:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:27.998 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-09-16T12:41:27.999 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:27.999 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.2 M 2024-09-16T12:41:27.999 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-09-16T12:41:27.999 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:28.209 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 26 kB 00:00 2024-09-16T12:41:28.234 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 129 kB/s | 19 kB 00:00 2024-09-16T12:41:28.259 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 274 kB/s | 47 kB 00:00 2024-09-16T12:41:28.326 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-16T12:41:28.351 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T12:41:28.377 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T12:41:28.435 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 386 kB/s | 32 kB 00:00 2024-09-16T12:41:28.452 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 227 kB/s | 14 kB 00:00 2024-09-16T12:41:28.477 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T12:41:28.494 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T12:41:28.495 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T12:41:28.495 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T12:41:28.495 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T12:41:28.495 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T12:41:28.519 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T12:41:28.544 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 886 kB/s | 59 kB 00:00 2024-09-16T12:41:28.570 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.9 MB/s | 90 kB 00:00 2024-09-16T12:41:28.612 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-16T12:41:28.616 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:28.616 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.6 MB/s | 2.2 MB 00:00 2024-09-16T12:41:28.685 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:28.702 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:28.702 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:28.793 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:28.795 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T12:41:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T12:41:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T12:41:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T12:41:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T12:41:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:28.934 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:29.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-09-16T12:41:29.501 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T12:41:29.562 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T12:41:29.619 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T12:41:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T12:41:29.697 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T12:41:29.799 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T12:41:29.909 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T12:41:29.914 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:29.965 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T12:41:30.049 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T12:41:30.050 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T12:41:30.051 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:30.052 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:30.052 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:30.052 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-09-16T12:41:30.052 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:30.052 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.2 M 2024-09-16T12:41:30.053 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-09-16T12:41:30.053 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:30.252 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 167 kB/s | 19 kB 00:00 2024-09-16T12:41:30.277 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 183 kB/s | 26 kB 00:00 2024-09-16T12:41:30.302 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 288 kB/s | 47 kB 00:00 2024-09-16T12:41:30.403 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-16T12:41:30.428 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-16T12:41:30.453 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T12:41:30.504 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-16T12:41:30.537 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 169 kB/s | 14 kB 00:00 2024-09-16T12:41:30.562 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-09-16T12:41:30.596 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-16T12:41:30.621 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-16T12:41:30.647 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T12:41:30.680 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T12:41:30.705 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-16T12:41:30.708 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:30.708 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T12:41:30.729 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T12:41:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T12:41:30.771 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:30.785 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:30.785 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:30.808 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T12:41:30.832 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T12:41:30.876 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:30.876 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:31.381 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T12:41:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T12:41:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T12:41:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T12:41:31.609 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T12:41:31.610 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T12:41:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T12:41:31.687 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T12:41:31.724 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T12:41:31.765 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T12:41:31.876 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T12:41:31.931 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T12:41:31.932 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T12:41:31.933 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T12:41:31.933 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T12:41:31.933 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:31.933 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:31.977 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T12:41:32.025 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T12:41:32.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T12:41:32.733 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T12:41:32.773 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:17 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:32.781 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T12:41:32.823 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T12:41:32.877 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T12:41:32.901 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T12:41:32.907 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:41:32.908 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-09-16T12:41:32.909 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:32.910 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 7.0 M 2024-09-16T12:41:32.910 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-09-16T12:41:32.910 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:33.163 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 632 kB/s | 124 kB 00:00 2024-09-16T12:41:33.238 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.2 MB/s | 548 kB 00:00 2024-09-16T12:41:33.263 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T12:41:33.297 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T12:41:33.614 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 384 kB/s | 248 kB 00:00 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T12:41:33.670 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T12:41:33.671 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T12:41:33.671 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T12:41:33.671 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T12:41:33.991 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T12:41:33.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T12:41:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T12:41:33.993 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:33.993 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:34.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T12:41:34.825 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:18 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:34.959 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:34.959 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:41:34.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T12:41:34.961 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T12:41:34.961 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T12:41:34.961 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T12:41:34.961 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T12:41:34.961 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:34.962 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:34.962 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:34.962 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-09-16T12:41:34.962 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:34.963 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 7.0 M 2024-09-16T12:41:34.963 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-09-16T12:41:34.963 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:35.340 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-16T12:41:35.374 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 563 kB/s | 124 kB 00:00 2024-09-16T12:41:35.432 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-16T12:41:35.466 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T12:41:35.558 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-16T12:41:35.591 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-09-16T12:41:35.594 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:35.594 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-16T12:41:35.664 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:35.675 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:35.675 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:35.773 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:35.775 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:36.076 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:36.249 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T12:41:36.310 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T12:41:36.647 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.6 MB/s | 6.0 MB 00:03 2024-09-16T12:41:36.649 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:36.650 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.9 MB/s | 7.0 MB 00:03 2024-09-16T12:41:36.723 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:36.733 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:36.734 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:36.831 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:36.831 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:36.840 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T12:41:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T12:41:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T12:41:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T12:41:37.140 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:37.303 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T12:41:37.365 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T12:41:37.851 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T12:41:37.851 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T12:41:37.851 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T12:41:37.852 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T12:41:37.852 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T12:41:37.852 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T12:41:37.887 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T12:41:37.967 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T12:41:38.023 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T12:41:38.053 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T12:41:38.129 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T12:41:38.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T12:41:38.130 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:38.130 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:38.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-09-16T12:41:38.767 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T12:41:38.767 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T12:41:38.767 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T12:41:38.767 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T12:41:38.767 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T12:41:38.768 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T12:41:38.981 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:39.033 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:39.114 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:39.115 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-09-16T12:41:39.116 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:39.116 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 893 k 2024-09-16T12:41:39.116 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.6 M 2024-09-16T12:41:39.116 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:39.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-09-16T12:41:39.393 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 536 kB/s | 80 kB 00:00 2024-09-16T12:41:39.535 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-16T12:41:39.535 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:39.536 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-16T12:41:39.548 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:39.569 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:39.569 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:39.647 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:39.648 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:39.837 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:39.849 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:39.974 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T12:41:39.982 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-16T12:41:39.983 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 893 k 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.6 M 2024-09-16T12:41:39.984 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:40.051 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T12:41:40.356 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 691 kB/s | 80 kB 00:00 2024-09-16T12:41:40.540 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-16T12:41:40.541 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:40.541 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-16T12:41:40.553 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:40.575 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:40.575 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:40.655 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:40.655 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T12:41:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T12:41:40.853 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:40.995 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:40.996 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T12:41:41.064 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T12:41:41.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-09-16T12:41:41.766 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T12:41:41.766 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T12:41:41.884 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:42.018 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:42.019 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 270 k 2024-09-16T12:41:42.020 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.3 M 2024-09-16T12:41:42.020 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:42.042 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T12:41:42.043 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:42.043 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:42.043 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T12:41:42.043 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:42.043 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:42.248 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-16T12:41:42.248 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:42.249 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-16T12:41:42.249 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:42.257 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:42.257 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:42.291 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:42.291 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:42.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2024-09-16T12:41:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:42.591 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:42.607 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:42.607 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:42.607 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T12:41:42.607 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:42.930 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:43.063 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 270 k 2024-09-16T12:41:43.064 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.3 M 2024-09-16T12:41:43.065 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:43.294 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-16T12:41:43.295 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:43.295 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-16T12:41:43.295 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:43.303 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:43.304 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:43.334 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:43.334 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:43.621 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:43.638 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:43.638 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:43.638 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T12:41:43.638 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:43.641 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:43.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-09-16T12:41:44.559 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T12:41:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:44.631 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T12:41:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:44.632 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:44.693 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:44.694 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-16T12:41:44.694 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:44.694 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 126 k 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 448 k 2024-09-16T12:41:44.695 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:44.894 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 635 kB/s | 126 kB 00:00 2024-09-16T12:41:44.895 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:44.895 INFO:teuthology.orchestra.run.smithi191.stdout:Total 631 kB/s | 126 kB 00:00 2024-09-16T12:41:44.895 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:44.900 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:44.900 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:44.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-09-16T12:41:44.962 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:44.962 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:45.117 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:45.216 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:45.485 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:45.620 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 126 k 2024-09-16T12:41:45.621 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 448 k 2024-09-16T12:41:45.622 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:45.808 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 675 kB/s | 126 kB 00:00 2024-09-16T12:41:45.808 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:45.808 INFO:teuthology.orchestra.run.smithi177.stdout:Total 669 kB/s | 126 kB 00:00 2024-09-16T12:41:45.808 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:45.814 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:45.814 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:45.876 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:45.877 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:45.906 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:46.025 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:46.122 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:46.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-09-16T12:41:46.571 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:46.734 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:46.822 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:46.823 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:46.823 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:46.823 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:46.823 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:46.823 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:46.827 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:46.870 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:46.871 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:46.871 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:46.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-09-16T12:41:47.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-09-16T12:41:47.535 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:47.643 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:47.667 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:47.668 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:47.669 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:47.669 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 25 k 2024-09-16T12:41:47.669 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 110 k 2024-09-16T12:41:47.669 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:47.736 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:47.781 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:47.782 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:47.782 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:47.814 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 172 kB/s | 25 kB 00:00 2024-09-16T12:41:47.815 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:47.815 INFO:teuthology.orchestra.run.smithi191.stdout:Total 170 kB/s | 25 kB 00:00 2024-09-16T12:41:47.815 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:47.818 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:47.819 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:47.820 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:47.820 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:47.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-09-16T12:41:47.904 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:48.003 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:48.339 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:48.425 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:48.558 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 25 k 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 110 k 2024-09-16T12:41:48.559 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:48.562 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:48.702 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 175 kB/s | 25 kB 00:00 2024-09-16T12:41:48.703 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:48.703 INFO:teuthology.orchestra.run.smithi177.stdout:Total 173 kB/s | 25 kB 00:00 2024-09-16T12:41:48.703 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:48.707 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:48.707 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:48.708 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:48.709 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:48.788 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:48.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-09-16T12:41:48.907 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:49.277 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:49.362 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:49.453 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:49.496 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:49.497 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:49.497 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:49.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-09-16T12:41:49.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-09-16T12:41:50.142 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:50.232 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:50.274 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:50.275 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:50.276 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:50.276 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:50.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-09-16T12:41:50.366 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:50.408 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:50.409 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:50.409 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:50.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-09-16T12:41:50.923 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:51.012 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:51.046 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:51.055 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:51.056 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:51.056 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:51.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-09-16T12:41:51.136 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:51.179 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:51.180 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:51.180 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:51.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-09-16T12:41:51.700 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:51.791 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:51.828 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:51.833 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:51.834 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:51.834 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:51.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-09-16T12:41:51.919 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:51.961 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:51.962 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:51.962 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:52.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-09-16T12:41:52.473 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:52.563 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:52.603 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:52.606 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:52.607 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:52.607 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:52.694 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:52.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-09-16T12:41:52.736 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:52.738 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:52.738 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:52.816 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-09-16T12:41:53.275 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:53.364 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:53.378 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:53.406 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:53.407 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:41:53.407 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:53.469 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:53.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-09-16T12:41:53.511 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:53.512 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:53.512 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:53.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-09-16T12:41:54.051 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:54.160 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:54.185 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:54.185 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:54.185 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 82 k 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 222 k 2024-09-16T12:41:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:54.250 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:41:54.292 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:54.293 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:41:54.293 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:54.376 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 433 kB/s | 82 kB 00:00 2024-09-16T12:41:54.376 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:54.376 INFO:teuthology.orchestra.run.smithi191.stdout:Total 430 kB/s | 82 kB 00:00 2024-09-16T12:41:54.377 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:54.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-09-16T12:41:54.383 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:54.383 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:54.445 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:54.446 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:54.586 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:54.689 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:54.949 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:55.054 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:55.083 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:55.083 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:55.083 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-16T12:41:55.083 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:55.083 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 82 k 2024-09-16T12:41:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 222 k 2024-09-16T12:41:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:55.280 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 419 kB/s | 82 kB 00:00 2024-09-16T12:41:55.281 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:55.281 INFO:teuthology.orchestra.run.smithi177.stdout:Total 416 kB/s | 82 kB 00:00 2024-09-16T12:41:55.281 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:55.288 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:55.288 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:55.304 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:55.305 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:55.305 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:55.305 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:55.305 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:55.305 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:55.350 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:55.350 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:55.492 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:55.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-09-16T12:41:55.586 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:55.980 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:56.146 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:56.244 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:56.284 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:56.284 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-16T12:41:56.284 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:56.284 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.9 M 2024-09-16T12:41:56.285 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 12 M 2024-09-16T12:41:56.286 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:56.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-09-16T12:41:56.620 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-16T12:41:56.621 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:56.621 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-16T12:41:56.621 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:56.714 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:56.714 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:57.115 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:41:57.152 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:57.230 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:57.250 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:41:57.250 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:57.250 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:41:57.251 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:57.252 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.9 M 2024-09-16T12:41:57.252 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 12 M 2024-09-16T12:41:57.252 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:41:57.256 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:57.592 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-09-16T12:41:57.592 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:57.593 INFO:teuthology.orchestra.run.smithi177.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-16T12:41:57.593 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:41:57.601 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:41:57.601 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:41:57.681 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:41:57.682 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:41:58.114 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:41:58.168 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:41:58.201 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:58.355 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:41:58.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-09-16T12:41:59.205 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:41:59.306 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:41:59.306 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:59.306 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:41:59.307 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:41:59.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:41:59.307 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:41:59.342 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:41:59.342 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:59.342 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-16T12:41:59.343 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:41:59.344 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 165 k 2024-09-16T12:41:59.344 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 477 k 2024-09-16T12:41:59.344 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-16T12:41:59.553 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 806 kB/s | 165 kB 00:00 2024-09-16T12:41:59.553 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:41:59.554 INFO:teuthology.orchestra.run.smithi191.stdout:Total 800 kB/s | 165 kB 00:00 2024-09-16T12:41:59.554 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-16T12:41:59.558 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-16T12:41:59.558 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-16T12:41:59.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-09-16T12:41:59.621 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-16T12:41:59.622 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-16T12:41:59.794 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-16T12:41:59.904 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:00.155 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:42:00.290 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-16T12:42:00.291 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:00.292 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 165 k 2024-09-16T12:42:00.292 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 477 k 2024-09-16T12:42:00.292 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-16T12:42:00.333 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:00.495 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 811 kB/s | 165 kB 00:00 2024-09-16T12:42:00.496 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-16T12:42:00.496 INFO:teuthology.orchestra.run.smithi177.stdout:Total 805 kB/s | 165 kB 00:00 2024-09-16T12:42:00.496 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-16T12:42:00.504 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-16T12:42:00.504 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-16T12:42:00.568 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-16T12:42:00.568 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-16T12:42:00.645 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:00.646 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:00.646 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-16T12:42:00.646 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:42:00.646 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:00.646 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:42:00.719 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-16T12:42:00.818 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:00.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-09-16T12:42:01.240 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:01.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T12:42:01.495 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:01.495 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-16T12:42:01.495 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T12:42:01.495 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:01.496 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:42:01.500 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 12:40:16 PM UTC. 2024-09-16T12:42:01.591 INFO:teuthology.orchestra.run.smithi191.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:42:01.636 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-16T12:42:01.637 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-16T12:42:01.637 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-16T12:42:01.733 DEBUG:teuthology.parallel:result is None 2024-09-16T12:42:01.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-09-16T12:42:02.358 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 12:40:15 PM UTC. 2024-09-16T12:42:02.449 INFO:teuthology.orchestra.run.smithi177.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T12:42:02.493 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-16T12:42:02.494 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-16T12:42:02.494 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-16T12:42:02.576 DEBUG:teuthology.parallel:result is None 2024-09-16T12:42:02.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:42:02.738 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T12:42:02.758 INFO:teuthology.orchestra.run.smithi177.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T12:42:02.759 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T12:42:02.759 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T12:42:02.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:42:02.904 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T12:42:02.929 INFO:teuthology.orchestra.run.smithi191.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T12:42:02.930 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T12:42:02.930 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T12:42:02.932 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T12:42:02.932 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:02.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T12:42:02.957 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:02.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T12:42:03.000 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T12:42:03.000 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:03.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T12:42:03.025 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T12:42:03.089 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:03.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T12:42:03.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T12:42:03.184 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T12:42:03.184 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:03.184 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T12:42:03.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T12:42:03.273 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:03.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T12:42:03.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T12:42:03.367 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T12:42:03.368 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:03.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T12:42:03.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T12:42:03.457 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:03.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T12:42:03.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T12:42:03.551 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T12:42:03.631 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T12:42:03.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-09-16T12:42:03.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-09-16T12:42:03.660 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T12:42:03.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T12:42:03.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T12:42:03.726 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T12:42:03.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T12:42:03.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T12:42:03.804 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-16T12:42:03.805 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T12:42:03.805 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907268', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxyY0W90XibMaBvgOmsGA3hrF1PoI912Y620dP5EBwCB/QPKP38POMBenu47QIbWrikdBMiN7zQN9uouLhvyPM=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxh05cLO93mtqfeS+mekZEU0jIRv4jztc6U/HIu0gS4gTbwL2YVsDDAvw98PX6JMboE2O1LXMtwgSZyWKhIMIg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T12:42:03.805 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T12:42:03.831 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T12:42:03.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:42:03.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:42:03.913 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:03.913 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T12:42:03.964 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T12:42:03.964 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-09-16T12:42:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T12:42:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 12:41:58.467066883 +0000 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 12:39:35.476369223 +0000 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 12:39:35.476369223 +0000 2024-09-16T12:42:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-16 12:39:35.476369223 +0000 2024-09-16T12:42:04.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T12:42:04.084 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T12:42:04.084 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T12:42:04.084 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000201033 s, 2.5 MB/s 2024-09-16T12:42:04.085 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T12:42:04.142 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-09-16T12:42:04.196 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 12:41:58.467066883 +0000 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 12:39:35.818359679 +0000 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 12:39:35.818359679 +0000 2024-09-16T12:42:04.197 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-16 12:39:35.818359679 +0000 2024-09-16T12:42:04.197 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T12:42:04.260 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T12:42:04.260 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T12:42:04.260 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000214169 s, 2.4 MB/s 2024-09-16T12:42:04.261 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T12:42:04.318 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 12:41:58.468066855 +0000 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 12:39:36.133350888 +0000 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 12:39:36.133350888 +0000 2024-09-16T12:42:04.373 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-16 12:39:36.133350888 +0000 2024-09-16T12:42:04.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T12:42:04.435 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T12:42:04.435 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T12:42:04.436 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000164612 s, 3.1 MB/s 2024-09-16T12:42:04.437 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T12:42:04.493 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 12:41:58.468066855 +0000 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 12:39:36.457341847 +0000 2024-09-16T12:42:04.549 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 12:39:36.457341847 +0000 2024-09-16T12:42:04.550 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-16 12:39:36.457341847 +0000 2024-09-16T12:42:04.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T12:42:04.611 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T12:42:04.612 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T12:42:04.612 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000208172 s, 2.5 MB/s 2024-09-16T12:42:04.613 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T12:42:04.669 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:04.669 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:42:04.669 INFO:tasks.ceph:Generating config... 2024-09-16T12:42:04.670 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T12:42:04.670 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T12:42:04.670 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T12:42:04.670 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T12:42:04.670 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:04.671 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:04.671 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T12:42:04.671 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T12:42:04.671 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T12:42:04.671 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T12:42:04.672 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T12:42:04.672 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T12:42:04.672 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:04.673 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T12:42:04.673 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T12:42:04.673 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T12:42:04.673 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-16T12:42:04.673 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T12:42:04.673 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-16T12:42:04.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-16T12:42:04.780 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/cluster1.keyring 2024-09-16T12:42:04.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-16T12:42:04.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:42:04.886 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.177')] 2024-09-16T12:42:04.886 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.177', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T12:42:04.887 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:04.887 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:42:04.941 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.177 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:05.017 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:05.017 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: generated fsid a00e16fb-57d0-4f62-8d54-577a0bafcdb7 2024-09-16T12:42:05.017 INFO:teuthology.orchestra.run.smithi177.stdout:setting min_mon_release = octopus 2024-09-16T12:42:05.017 INFO:teuthology.orchestra.run.smithi177.stdout:epoch 0 2024-09-16T12:42:05.017 INFO:teuthology.orchestra.run.smithi177.stdout:fsid a00e16fb-57d0-4f62-8d54-577a0bafcdb7 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:last_changed 2024-09-16T12:42:05.016533+0000 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:created 2024-09-16T12:42:05.016533+0000 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:min_mon_release 15 (octopus) 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:election_strategy: 1 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:0: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.a 2024-09-16T12:42:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-16T12:42:05.019 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:42:05.074 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID a00e16fb-57d0-4f62-8d54-577a0bafcdb7... 2024-09-16T12:42:05.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T12:42:05.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-09-16T12:42:05.156 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-09-16T12:42:05.157 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-09-16T12:42:05.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = 172.21.15.177 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = a00e16fb-57d0-4f62-8d54-577a0bafcdb7 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.159 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:42:05.160 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-09-16T12:42:05.161 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: osd memory target = 1610612736 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T12:42:05.162 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-09-16T12:42:05.163 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:05.164 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: debug rbd = 20 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: debug rbd_mirror = 15 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: log to stderr = False 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: rbd default features = 125 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: rbd mirror image policy type = simple 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.0] 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.165 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.1] 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.2] 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.3] 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.166 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-16T12:42:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:42:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-16T12:42:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = 172.21.15.177 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = a00e16fb-57d0-4f62-8d54-577a0bafcdb7 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-16T12:42:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: osd memory target = 1610612736 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:42:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:42:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd = 20 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd_mirror = 15 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: log to stderr = False 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: rbd default features = 125 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: rbd mirror image policy type = simple 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.176 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-16T12:42:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:05.179 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-16T12:42:05.179 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-16T12:42:05.267 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T12:42:05.268 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:05.268 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-16T12:42:05.283 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:05.283 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-16T12:42:05.338 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:42:05.338 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:05.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T12:42:05.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:42:05.412 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:05.412 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:05.468 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:42:05.468 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:05.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T12:42:05.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:42:05.510 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:05.510 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:05.567 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T12:42:05.567 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T12:42:05.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T12:42:05.614 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T12:42:05.616 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T12:42:05.617 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T12:42:05.617 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-16T12:42:05.654 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-16T12:42:05.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T12:42:05.714 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T12:42:05.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T12:42:05.766 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T12:42:05.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T12:42:05.857 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T12:42:05.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T12:42:05.947 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T12:42:05.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T12:42:06.037 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T12:42:06.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T12:42:06.125 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T12:42:06.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T12:42:06.215 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T12:42:06.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T12:42:06.304 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T12:42:06.316 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T12:42:06.316 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:42:06.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-16T12:42:06.342 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:06.343 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-16T12:42:06.343 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:42:06.343 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T12:42:06.415 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:06.416 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:06.905 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T12:42:06.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T12:42:06.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-16T12:42:07.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-16T12:42:07.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-16T12:42:07.109 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:07.109 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-16T12:42:07.110 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:42:07.110 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:07.183 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:07.184 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:07.184 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:07.184 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:07.638 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T12:42:07.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T12:42:07.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-16T12:42:07.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-16T12:42:07.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-16T12:42:07.858 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:07.858 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-16T12:42:07.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:42:07.858 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:07.931 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:07.932 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:08.411 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T12:42:08.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T12:42:08.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-16T12:42:08.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-16T12:42:08.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:08.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.631+0000 7f411272a440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-16T12:42:08.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.634+0000 7f411272a440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-16T12:42:08.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.634+0000 7f411272a440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:42:08.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.634+0000 7f411272a440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:42:08.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.634+0000 7f411272a440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:42:08.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:08.636+0000 7f411272a440 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-16T12:42:09.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:09.169+0000 7f411272a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:09.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:09.929+0000 7f411272a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:10.697 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:10.695+0000 7f411272a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:11.487 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-16T12:42:11.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.600+0000 7f182552b440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-16T12:42:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.600+0000 7f182552b440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-16T12:42:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.600+0000 7f182552b440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:42:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.600+0000 7f182552b440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:42:11.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.600+0000 7f182552b440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:42:11.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:11.601+0000 7f182552b440 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-16T12:42:12.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:12.125+0000 7f182552b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:12.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:12.887+0000 7f182552b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:13.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:13.650+0000 7f182552b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:14.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-16T12:42:14.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:14.557 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.555+0000 7fac6e64e440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-16T12:42:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.556+0000 7fac6e64e440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-16T12:42:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.556+0000 7fac6e64e440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:42:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.556+0000 7fac6e64e440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:42:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.556+0000 7fac6e64e440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:42:14.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:14.556+0000 7fac6e64e440 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-16T12:42:15.083 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:15.082+0000 7fac6e64e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:15.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:15.847+0000 7fac6e64e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:16.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T12:42:16.609+0000 7fac6e64e440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:17.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-16T12:42:17.440 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T12:42:17.440 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:17.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-16T12:42:17.508 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:17.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-16T12:42:17.575 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:17.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-16T12:42:17.641 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:17.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-16T12:42:17.707 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:17.707 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-16T12:42:17.763 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:17.763 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-16T12:42:17.782 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:17.783 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-16T12:42:17.841 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:17.841 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-16T12:42:17.899 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:17.899 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-16T12:42:17.957 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:17.957 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-16T12:42:18.015 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:18.015 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-16T12:42:18.072 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:18.072 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-16T12:42:18.129 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:18.129 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-16T12:42:18.186 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T12:42:18.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hmZzWRJBAA7mg8ToMoYHhHMepi8v/qwA== 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCgJ+hmEWOrJRAAGBf0+E0P98qPIa/ZPHi6zA== 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCjJ+hmtv3VIxAAM8+FnSVgckFE+aS1kTfbeA== 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCmJ+hmI6ExIRAAT5G3A1LtFCsp8maLqaY9nw== 2024-09-16T12:42:18.212 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hmk6ntJhAA9pLcwNU1EW8sb3jbbYKZPQ== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hmbpKEKhAA6mcNGZ8PfeGGhM5ieb4bNQ== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.0] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hm9/adLRAAnoyOJ1I+slDv6XTuXIu0ig== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.1] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hmfQEOMxAAaFNPZSCQE21Gkn5yF4GXFw== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.2] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCdJ+hm1KZzOBAADgMfbvxcAzBGFXRTJ3sUnQ== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.3] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCeJ+hmjIUpAhAANvuYmuDtLLFqiJggSezT3A== 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.4] 2024-09-16T12:42:18.213 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCeJ+hmDGp1BxAAHJVYQhzwTzevE/Zue8QcFg== 2024-09-16T12:42:18.214 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.5] 2024-09-16T12:42:18.214 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCeJ+hmJmrLDBAAnGrNWNKioCK4Sx3m61UPiQ== 2024-09-16T12:42:18.214 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.6] 2024-09-16T12:42:18.214 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQCeJ+hmYCgWEhAAoznsZ2ysTgv28ybWV1/ckw== 2024-09-16T12:42:18.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T12:42:18.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:18.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:18.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:18.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.640 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:18.885 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T12:42:18.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-16T12:42:18.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-16T12:42:18.999 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-16T12:42:19.024 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:42:19.079 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-16T12:42:19.079 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-16T12:42:19.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-16T12:42:19.121 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-16T12:42:19.121 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-16T12:42:19.121 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-16T12:42:19.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-16T12:42:19.123 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-16T12:42:19.124 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T12:42:19.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-16T12:42:19.288 INFO:teuthology.orchestra.run.smithi177.stderr:adjusted tunables profile to default 2024-09-16T12:42:19.299 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-16T12:42:19.299 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:19.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-16T12:42:19.323 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:19.323 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-16T12:42:19.388 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:19.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-16T12:42:19.453 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 osd new 75458b1f-ab53-455c-b7b7-e4581435a54c 0 2024-09-16T12:42:19.491 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:19.489+0000 7ffb36bbdec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T12:42:19.532 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:19.530+0000 7ffb36bbdec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T12:42:19.585 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:19.583+0000 7ffb36bbdec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T12:42:19.661 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-09-16T12:42:19.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 osd new 295d1942-3818-4b37-a350-593db1429c7f 1 2024-09-16T12:42:19.729 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:19.727+0000 7ffb36bbdec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:42:19.842 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:19.840+0000 7ffb36bbdec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T12:42:19.882 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-09-16T12:42:19.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 osd new 11e62303-e15c-4bc3-8531-080922db8aa7 2 2024-09-16T12:42:20.103 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-09-16T12:42:20.114 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-16T12:42:20.114 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-16T12:42:20.116 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-16T12:42:20.116 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-16T12:42:20.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-16T12:42:20.118 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-16T12:42:20.118 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-16T12:42:20.118 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-16T12:42:20.120 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-16T12:42:20.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:20.137 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:20.136+0000 7ffb36bbdec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T12:42:20.199 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:20.197+0000 7ffb36bbdec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T12:42:20.304 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:20.304 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":5,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:20.100535+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:20.315 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-16T12:42:20.316 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T12:42:20.340 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:20.338+0000 7ffb36bbdec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T12:42:20.619 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:20.713 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:42:20.711+0000 7f6079b84440 -1 Falling back to public interface 2024-09-16T12:42:20.714 INFO:tasks.ceph.cluster1.osd.0.smithi177.stderr:2024-09-16T12:42:20.712+0000 7f93c7b66440 -1 Falling back to public interface 2024-09-16T12:42:20.725 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T12:42:20.723+0000 7feb4ce08440 -1 Falling back to public interface 2024-09-16T12:42:20.774 INFO:teuthology.misc.health.smithi177.stdout: 2024-09-16T12:42:20.774 INFO:teuthology.misc.health.smithi177.stdout:{"epoch":5,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:20.100535+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:20.783 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:42:21.270 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.268+0000 7ffb36bbdec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T12:42:21.321 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.319+0000 7ffb36bbdec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T12:42:21.682 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.680+0000 7ffb36bbdec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T12:42:21.762 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.760+0000 7ffb36bbdec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T12:42:21.853 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.851+0000 7ffb36bbdec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:42:21.897 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.895+0000 7ffb36bbdec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T12:42:21.990 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:21.988+0000 7ffb36bbdec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T12:42:22.041 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.040+0000 7ffb36bbdec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T12:42:22.086 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.084+0000 7ffb36bbdec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T12:42:22.179 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.177+0000 7ffb36bbdec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T12:42:22.228 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.226+0000 7ffb36bbdec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T12:42:22.277 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.275+0000 7ffb36bbdec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T12:42:22.342 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.340+0000 7ffb36bbdec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T12:42:22.386 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.384+0000 7ffb36bbdec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T12:42:22.429 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:22.427+0000 7ffb36bbdec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T12:42:23.041 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:23.039+0000 7ffb36bbdec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T12:42:24.048 INFO:tasks.ceph.cluster1.mgr.x.smithi177.stderr:2024-09-16T12:42:24.046+0000 7ffb1b289640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T12:42:24.238 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:42:24.236+0000 7f6079b84440 -1 osd.2 0 log_to_monitors true 2024-09-16T12:42:24.238 INFO:tasks.ceph.cluster1.osd.0.smithi177.stderr:2024-09-16T12:42:24.236+0000 7f93c7b66440 -1 osd.0 0 log_to_monitors true 2024-09-16T12:42:24.265 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T12:42:24.263+0000 7feb4ce08440 -1 osd.1 0 log_to_monitors true 2024-09-16T12:42:27.086 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:27.494 INFO:teuthology.misc.health.smithi177.stdout: 2024-09-16T12:42:27.494 INFO:teuthology.misc.health.smithi177.stdout:{"epoch":7,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:26.047613+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:27.509 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:42:27.745 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T12:42:27.743+0000 7feb475f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T12:42:27.752 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T12:42:27.750+0000 7feb405e8640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:42:27.799 INFO:tasks.ceph.cluster1.osd.0.smithi177.stderr:2024-09-16T12:42:27.797+0000 7f93c25f6640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T12:42:27.806 INFO:tasks.ceph.cluster1.osd.0.smithi177.stderr:2024-09-16T12:42:27.804+0000 7f93bb5e8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:42:27.856 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:42:27.854+0000 7f60745f6640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T12:42:27.861 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:42:27.859+0000 7f606d5e8640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:42:33.812 DEBUG:teuthology.orchestra.run.smithi177:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:34.077 INFO:teuthology.misc.health.smithi177.stdout: 2024-09-16T12:42:34.077 INFO:teuthology.misc.health.smithi177.stdout:{"epoch":11,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:32.060345+0000","last_up_change":"2024-09-16T12:42:28.747107+0000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:42:29.075336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":32046},{"type":"v1","addr":"172.21.15.177:6809","nonce":32046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":32046},{"type":"v1","addr":"172.21.15.177:6811","nonce":32046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":32046},{"type":"v1","addr":"172.21.15.177:6815","nonce":32046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":32046},{"type":"v1","addr":"172.21.15.177:6813","nonce":32046}]},"public_addr":"172.21.15.177:6809/32046","cluster_addr":"172.21.15.177:6811/32046","heartbeat_back_addr":"172.21.15.177:6815/32046","heartbeat_front_addr":"172.21.15.177:6813/32046","state":["exists","up"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":32050},{"type":"v1","addr":"172.21.15.177:6817","nonce":32050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":32050},{"type":"v1","addr":"172.21.15.177:6819","nonce":32050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":32050},{"type":"v1","addr":"172.21.15.177:6823","nonce":32050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":32050},{"type":"v1","addr":"172.21.15.177:6821","nonce":32050}]},"public_addr":"172.21.15.177:6817/32050","cluster_addr":"172.21.15.177:6819/32050","heartbeat_back_addr":"172.21.15.177:6823/32050","heartbeat_front_addr":"172.21.15.177:6821/32050","state":["exists","up"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":32047},{"type":"v1","addr":"172.21.15.177:6801","nonce":32047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":32047},{"type":"v1","addr":"172.21.15.177:6803","nonce":32047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":32047},{"type":"v1","addr":"172.21.15.177:6807","nonce":32047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":32047},{"type":"v1","addr":"172.21.15.177:6805","nonce":32047}]},"public_addr":"172.21.15.177:6801/32047","cluster_addr":"172.21.15.177:6803/32047","heartbeat_back_addr":"172.21.15.177:6807/32047","heartbeat_front_addr":"172.21.15.177:6805/32047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.214807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.219527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.236130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:34.088 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T12:42:34.088 INFO:tasks.ceph:Creating RBD pool 2024-09-16T12:42:34.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-16T12:42:35.077 INFO:teuthology.orchestra.run.smithi177.stderr:pool 'rbd' created 2024-09-16T12:42:35.093 DEBUG:teuthology.orchestra.run.smithi177:> rbd --cluster cluster1 pool init rbd 2024-09-16T12:42:38.094 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-16T12:42:38.095 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-16T12:42:38.095 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-16T12:42:38.095 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-16T12:42:38.095 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T12:42:38.406 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:38.417 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":31709},{"type":"v1","addr":"172.21.15.177:6825","nonce":31709}]},"active_addr":"172.21.15.177:6825/31709","active_change":"2024-09-16T12:42:23.041430+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1798935043}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3913321122}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3145522593}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":548718814}]}]} 2024-09-16T12:42:38.419 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-16T12:42:38.419 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-16T12:42:38.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:38.679 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:38.679 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":15,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:38.080994+0000","last_up_change":"2024-09-16T12:42:28.747107+0000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:42:29.075336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:42:34.363093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":32046},{"type":"v1","addr":"172.21.15.177:6809","nonce":32046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":32046},{"type":"v1","addr":"172.21.15.177:6811","nonce":32046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":32046},{"type":"v1","addr":"172.21.15.177:6815","nonce":32046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":32046},{"type":"v1","addr":"172.21.15.177:6813","nonce":32046}]},"public_addr":"172.21.15.177:6809/32046","cluster_addr":"172.21.15.177:6811/32046","heartbeat_back_addr":"172.21.15.177:6815/32046","heartbeat_front_addr":"172.21.15.177:6813/32046","state":["exists","up"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":32050},{"type":"v1","addr":"172.21.15.177:6817","nonce":32050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":32050},{"type":"v1","addr":"172.21.15.177:6819","nonce":32050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":32050},{"type":"v1","addr":"172.21.15.177:6823","nonce":32050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":32050},{"type":"v1","addr":"172.21.15.177:6821","nonce":32050}]},"public_addr":"172.21.15.177:6817/32050","cluster_addr":"172.21.15.177:6819/32050","heartbeat_back_addr":"172.21.15.177:6823/32050","heartbeat_front_addr":"172.21.15.177:6821/32050","state":["exists","up"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":32047},{"type":"v1","addr":"172.21.15.177:6801","nonce":32047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":32047},{"type":"v1","addr":"172.21.15.177:6803","nonce":32047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":32047},{"type":"v1","addr":"172.21.15.177:6807","nonce":32047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":32047},{"type":"v1","addr":"172.21.15.177:6805","nonce":32047}]},"public_addr":"172.21.15.177:6801/32047","cluster_addr":"172.21.15.177:6803/32047","heartbeat_back_addr":"172.21.15.177:6807/32047","heartbeat_front_addr":"172.21.15.177:6805/32047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.214807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.219527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.236130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:38.691 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-16T12:42:38.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:42:38.951 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:38.951 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":15,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T12:42:38.080994+0000","last_up_change":"2024-09-16T12:42:28.747107+0000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:42:29.075336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:42:34.363093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":32046},{"type":"v1","addr":"172.21.15.177:6809","nonce":32046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":32046},{"type":"v1","addr":"172.21.15.177:6811","nonce":32046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":32046},{"type":"v1","addr":"172.21.15.177:6815","nonce":32046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":32046},{"type":"v1","addr":"172.21.15.177:6813","nonce":32046}]},"public_addr":"172.21.15.177:6809/32046","cluster_addr":"172.21.15.177:6811/32046","heartbeat_back_addr":"172.21.15.177:6815/32046","heartbeat_front_addr":"172.21.15.177:6813/32046","state":["exists","up"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":32050},{"type":"v1","addr":"172.21.15.177:6817","nonce":32050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":32050},{"type":"v1","addr":"172.21.15.177:6819","nonce":32050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":32050},{"type":"v1","addr":"172.21.15.177:6823","nonce":32050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":32050},{"type":"v1","addr":"172.21.15.177:6821","nonce":32050}]},"public_addr":"172.21.15.177:6817/32050","cluster_addr":"172.21.15.177:6819/32050","heartbeat_back_addr":"172.21.15.177:6823/32050","heartbeat_front_addr":"172.21.15.177:6821/32050","state":["exists","up"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":32047},{"type":"v1","addr":"172.21.15.177:6801","nonce":32047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":32047},{"type":"v1","addr":"172.21.15.177:6803","nonce":32047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":32047},{"type":"v1","addr":"172.21.15.177:6807","nonce":32047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":32047},{"type":"v1","addr":"172.21.15.177:6805","nonce":32047}]},"public_addr":"172.21.15.177:6801/32047","cluster_addr":"172.21.15.177:6803/32047","heartbeat_back_addr":"172.21.15.177:6807/32047","heartbeat_front_addr":"172.21.15.177:6805/32047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.214807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.219527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.236130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:38.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-16T12:42:38.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-16T12:42:38.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-16T12:42:39.096 INFO:teuthology.orchestra.run.smithi177.stdout:34359738375 2024-09-16T12:42:39.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T12:42:39.109 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:39.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T12:42:39.111 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:39.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T12:42:39.375 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:39.388 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738373 for osd.0 2024-09-16T12:42:39.438 INFO:teuthology.orchestra.run.smithi177.stdout:34359738371 2024-09-16T12:42:39.445 INFO:teuthology.orchestra.run.smithi177.stdout:34359738371 2024-09-16T12:42:39.451 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.1 2024-09-16T12:42:39.457 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.2 2024-09-16T12:42:40.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T12:42:40.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T12:42:40.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T12:42:40.670 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:40.682 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738373 for osd.0 2024-09-16T12:42:40.744 INFO:teuthology.orchestra.run.smithi177.stdout:34359738371 2024-09-16T12:42:40.756 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.1 2024-09-16T12:42:40.781 INFO:teuthology.orchestra.run.smithi177.stdout:34359738371 2024-09-16T12:42:40.793 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.2 2024-09-16T12:42:41.684 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T12:42:41.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T12:42:41.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T12:42:41.970 INFO:teuthology.orchestra.run.smithi177.stdout:34359738375 2024-09-16T12:42:41.982 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.0 2024-09-16T12:42:41.983 DEBUG:teuthology.parallel:result is None 2024-09-16T12:42:42.036 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:42.048 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-09-16T12:42:42.048 DEBUG:teuthology.parallel:result is None 2024-09-16T12:42:42.074 INFO:teuthology.orchestra.run.smithi177.stdout:34359738373 2024-09-16T12:42:42.087 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-16T12:42:42.087 DEBUG:teuthology.parallel:result is None 2024-09-16T12:42:42.087 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T12:42:42.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T12:42:42.386 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:42.387 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T12:42:42.398 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T12:42:41.047202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893640,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221880,"statfs":{"total":289910292480,"available":288995205120,"internally_reserved":0,"allocated":1589248,"data_stored":1295299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.972457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089295+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T12:42:38.089295+0000","last_peered":"2024-09-16T12:42:38.089295+0000","last_clean":"2024-09-16T12:42:38.089295+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T12:42:38.089295+0000","last_undegraded":"2024-09-16T12:42:38.089295+0000","last_fullsized":"2024-09-16T12:42:38.089295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:11.040470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089562+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T12:42:38.089562+0000","last_peered":"2024-09-16T12:42:38.089562+0000","last_clean":"2024-09-16T12:42:38.089562+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T12:42:38.089562+0000","last_undegraded":"2024-09-16T12:42:38.089562+0000","last_fullsized":"2024-09-16T12:42:38.089562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:37.417191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089245+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T12:42:38.089245+0000","last_peered":"2024-09-16T12:42:38.089245+0000","last_clean":"2024-09-16T12:42:38.089245+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T12:42:38.089245+0000","last_undegraded":"2024-09-16T12:42:38.089245+0000","last_fullsized":"2024-09-16T12:42:38.089245+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:26.291723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.088874+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T12:42:38.088874+0000","last_peered":"2024-09-16T12:42:38.088874+0000","last_clean":"2024-09-16T12:42:38.088874+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T12:42:38.088874+0000","last_undegraded":"2024-09-16T12:42:38.088874+0000","last_fullsized":"2024-09-16T12:42:38.088874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:45.836097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.087391+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T12:42:38.087391+0000","last_peered":"2024-09-16T12:42:38.087391+0000","last_clean":"2024-09-16T12:42:38.087391+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T12:42:38.087391+0000","last_undegraded":"2024-09-16T12:42:38.087391+0000","last_fullsized":"2024-09-16T12:42:38.087391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:34.320798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.083841+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T12:42:38.083841+0000","last_peered":"2024-09-16T12:42:38.083841+0000","last_clean":"2024-09-16T12:42:38.083841+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T12:42:38.083841+0000","last_undegraded":"2024-09-16T12:42:38.083841+0000","last_fullsized":"2024-09-16T12:42:38.083841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.083811+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T12:42:38.083811+0000","last_peered":"2024-09-16T12:42:38.083811+0000","last_clean":"2024-09-16T12:42:38.083811+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T12:42:38.083811+0000","last_undegraded":"2024-09-16T12:42:38.083811+0000","last_fullsized":"2024-09-16T12:42:38.083811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.088821+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T12:42:38.088821+0000","last_peered":"2024-09-16T12:42:38.088821+0000","last_clean":"2024-09-16T12:42:38.088821+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T12:42:38.088821+0000","last_undegraded":"2024-09-16T12:42:38.088821+0000","last_fullsized":"2024-09-16T12:42:38.088821+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:51.045883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":80,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T12:42:37.085703+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T12:42:37.085703+0000","last_peered":"2024-09-16T12:42:37.085703+0000","last_clean":"2024-09-16T12:42:37.085703+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T12:42:37.085703+0000","last_undegraded":"2024-09-16T12:42:37.085703+0000","last_fullsized":"2024-09-16T12:42:37.085703+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.333013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":131072,"data_stored":38175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:42:42.399 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T12:42:42.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:42.654 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T12:42:42.665 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T12:42:41.047202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893640,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221880,"statfs":{"total":289910292480,"available":288995205120,"internally_reserved":0,"allocated":1589248,"data_stored":1295299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.972457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089295+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T12:42:38.089295+0000","last_peered":"2024-09-16T12:42:38.089295+0000","last_clean":"2024-09-16T12:42:38.089295+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T12:42:38.089295+0000","last_undegraded":"2024-09-16T12:42:38.089295+0000","last_fullsized":"2024-09-16T12:42:38.089295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:11.040470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089562+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T12:42:38.089562+0000","last_peered":"2024-09-16T12:42:38.089562+0000","last_clean":"2024-09-16T12:42:38.089562+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T12:42:38.089562+0000","last_undegraded":"2024-09-16T12:42:38.089562+0000","last_fullsized":"2024-09-16T12:42:38.089562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:37.417191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.089245+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T12:42:38.089245+0000","last_peered":"2024-09-16T12:42:38.089245+0000","last_clean":"2024-09-16T12:42:38.089245+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T12:42:38.089245+0000","last_undegraded":"2024-09-16T12:42:38.089245+0000","last_fullsized":"2024-09-16T12:42:38.089245+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:26.291723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.088874+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T12:42:38.088874+0000","last_peered":"2024-09-16T12:42:38.088874+0000","last_clean":"2024-09-16T12:42:38.088874+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T12:42:38.088874+0000","last_undegraded":"2024-09-16T12:42:38.088874+0000","last_fullsized":"2024-09-16T12:42:38.088874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:45.836097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.087391+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T12:42:38.087391+0000","last_peered":"2024-09-16T12:42:38.087391+0000","last_clean":"2024-09-16T12:42:38.087391+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T12:42:38.087391+0000","last_undegraded":"2024-09-16T12:42:38.087391+0000","last_fullsized":"2024-09-16T12:42:38.087391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:34.320798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.083841+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T12:42:38.083841+0000","last_peered":"2024-09-16T12:42:38.083841+0000","last_clean":"2024-09-16T12:42:38.083841+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T12:42:38.083841+0000","last_undegraded":"2024-09-16T12:42:38.083841+0000","last_fullsized":"2024-09-16T12:42:38.083841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.083811+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T12:42:38.083811+0000","last_peered":"2024-09-16T12:42:38.083811+0000","last_clean":"2024-09-16T12:42:38.083811+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T12:42:38.083811+0000","last_undegraded":"2024-09-16T12:42:38.083811+0000","last_fullsized":"2024-09-16T12:42:38.083811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:42:38.088821+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T12:42:38.088821+0000","last_peered":"2024-09-16T12:42:38.088821+0000","last_clean":"2024-09-16T12:42:38.088821+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T12:42:38.088821+0000","last_undegraded":"2024-09-16T12:42:38.088821+0000","last_fullsized":"2024-09-16T12:42:38.088821+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:51.045883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":80,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T12:42:37.085703+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T12:42:37.085703+0000","last_peered":"2024-09-16T12:42:37.085703+0000","last_clean":"2024-09-16T12:42:37.085703+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T12:42:37.085703+0000","last_undegraded":"2024-09-16T12:42:37.085703+0000","last_fullsized":"2024-09-16T12:42:37.085703+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.333013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":131072,"data_stored":38175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:42:42.666 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T12:42:42.666 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-16T12:42:42.666 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-16T12:42:42.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-16T12:42:42.953 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:42.953 INFO:teuthology.orchestra.run.smithi177.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T12:42:42.965 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-16T12:42:42.965 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T12:42:42.974 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-16T12:42:42.974 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T12:42:42.974 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907268', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxyY0W90XibMaBvgOmsGA3hrF1PoI912Y620dP5EBwCB/QPKP38POMBenu47QIbWrikdBMiN7zQN9uouLhvyPM=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxh05cLO93mtqfeS+mekZEU0jIRv4jztc6U/HIu0gS4gTbwL2YVsDDAvw98PX6JMboE2O1LXMtwgSZyWKhIMIg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T12:42:42.976 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T12:42:42.978 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T12:42:42.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:42:43.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:42:43.066 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:43.067 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T12:42:43.123 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T12:42:43.123 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-16 12:41:57.523220824 +0000 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-16 12:39:35.501044082 +0000 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-16 12:39:35.501044082 +0000 2024-09-16T12:42:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-16 12:39:35.501044082 +0000 2024-09-16T12:42:43.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T12:42:43.245 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-16T12:42:43.246 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-16T12:42:43.246 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000218218 s, 2.3 MB/s 2024-09-16T12:42:43.247 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T12:42:43.304 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-09-16T12:42:43.360 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-16 12:41:57.524220806 +0000 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-16 12:39:35.835038253 +0000 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-16 12:39:35.835038253 +0000 2024-09-16T12:42:43.361 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-16 12:39:35.835038253 +0000 2024-09-16T12:42:43.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T12:42:43.427 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-16T12:42:43.427 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-16T12:42:43.427 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000218565 s, 2.3 MB/s 2024-09-16T12:42:43.428 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T12:42:43.484 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-16 12:41:57.524220806 +0000 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-16 12:39:36.161032564 +0000 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-16 12:39:36.161032564 +0000 2024-09-16T12:42:43.540 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-16 12:39:36.161032564 +0000 2024-09-16T12:42:43.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T12:42:43.604 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-16T12:42:43.605 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-16T12:42:43.605 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000204333 s, 2.5 MB/s 2024-09-16T12:42:43.606 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T12:42:43.662 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-09-16T12:42:43.718 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-16 12:41:57.524220806 +0000 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-16 12:39:36.494026752 +0000 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-16 12:39:36.494026752 +0000 2024-09-16T12:42:43.719 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-16 12:39:36.494026752 +0000 2024-09-16T12:42:43.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T12:42:43.782 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-16T12:42:43.782 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-16T12:42:43.782 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000186185 s, 2.7 MB/s 2024-09-16T12:42:43.783 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T12:42:43.839 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:43.839 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:42:43.839 INFO:tasks.ceph:Generating config... 2024-09-16T12:42:43.840 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:43.841 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:43.841 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T12:42:43.842 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T12:42:43.842 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T12:42:43.842 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T12:42:43.842 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T12:42:43.842 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T12:42:43.842 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T12:42:43.842 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T12:42:43.842 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-16T12:42:43.843 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T12:42:43.843 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-16T12:42:43.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-16T12:42:43.915 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.keyring 2024-09-16T12:42:43.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-16T12:42:43.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:42:44.062 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.191')] 2024-09-16T12:42:44.062 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.191', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T12:42:44.063 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:44.063 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:42:44.117 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.191 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:44.193 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:44.193 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: generated fsid 15053695-24d8-4e94-adfc-94f39586fb95 2024-09-16T12:42:44.193 INFO:teuthology.orchestra.run.smithi191.stdout:setting min_mon_release = octopus 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:epoch 0 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:fsid 15053695-24d8-4e94-adfc-94f39586fb95 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:last_changed 2024-09-16T12:42:44.192622+0000 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:created 2024-09-16T12:42:44.192622+0000 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:min_mon_release 15 (octopus) 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:election_strategy: 1 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:0: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.a 2024-09-16T12:42:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-16T12:42:44.195 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:42:44.250 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 15053695-24d8-4e94-adfc-94f39586fb95... 2024-09-16T12:42:44.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T12:42:44.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-09-16T12:42:44.302 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:42:44.303 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-09-16T12:42:44.304 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = 172.21.15.191 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2024-09-16T12:42:44.305 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 15053695-24d8-4e94-adfc-94f39586fb95 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-09-16T12:42:44.306 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.307 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:42:44.308 INFO:teuthology.orchestra.run.smithi177.stdout: osd memory target = 1610612736 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T12:42:44.309 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:42:44.310 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: debug rbd = 20 2024-09-16T12:42:44.311 INFO:teuthology.orchestra.run.smithi177.stdout: debug rbd_mirror = 15 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: log to stderr = False 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: rbd default features = 125 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: rbd mirror image policy type = simple 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.0] 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.1] 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.2] 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.312 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.313 INFO:teuthology.orchestra.run.smithi177.stdout:[client.mirror.3] 2024-09-16T12:42:44.313 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.313 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.333 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:42:44.334 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:42:44.335 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = 172.21.15.191 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 15053695-24d8-4e94-adfc-94f39586fb95 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.336 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.337 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-16T12:42:44.338 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: osd memory target = 1610612736 2024-09-16T12:42:44.339 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-16T12:42:44.340 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:42:44.341 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd = 20 2024-09-16T12:42:44.342 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd_mirror = 15 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: log to stderr = False 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: rbd default features = 125 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: rbd mirror image policy type = simple 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.343 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-16T12:42:44.344 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:42:44.344 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:42:44.344 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-16T12:42:44.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-16T12:42:44.434 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T12:42:44.435 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:44.435 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-16T12:42:44.450 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:44.450 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-16T12:42:44.505 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T12:42:44.506 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:44.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T12:42:44.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:42:44.544 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:44.544 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:44.601 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:42:44.601 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:44.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T12:42:44.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:42:44.635 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:44.635 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:44.691 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T12:42:44.692 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T12:42:44.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T12:42:44.741 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T12:42:44.743 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T12:42:44.744 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T12:42:44.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-16T12:42:44.781 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-16T12:42:44.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T12:42:44.831 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T12:42:44.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T12:42:44.920 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T12:42:44.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T12:42:45.010 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T12:42:45.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T12:42:45.099 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T12:42:45.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T12:42:45.188 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T12:42:45.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T12:42:45.278 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T12:42:45.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T12:42:45.368 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T12:42:45.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T12:42:45.457 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T12:42:45.470 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T12:42:45.470 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:42:45.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-16T12:42:45.535 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:45.535 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-16T12:42:45.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:42:45.535 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:45.607 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:45.608 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:45.608 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:45.608 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:45.608 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:46.118 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-16T12:42:46.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-16T12:42:46.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-16T12:42:46.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-16T12:42:46.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-16T12:42:46.365 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:46.365 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-16T12:42:46.365 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:42:46.365 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:46.439 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:46.883 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-16T12:42:46.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-16T12:42:46.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-16T12:42:46.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-16T12:42:47.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-16T12:42:47.076 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:42:47.076 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-16T12:42:47.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:42:47.077 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:42:47.149 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:42:47.629 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-16T12:42:47.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-16T12:42:47.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-16T12:42:47.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-16T12:42:47.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:47.910 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.908+0000 7f4fa6608440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-16T12:42:47.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.915+0000 7f4fa6608440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-16T12:42:47.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.915+0000 7f4fa6608440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:42:47.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.915+0000 7f4fa6608440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:42:47.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.915+0000 7f4fa6608440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:42:47.921 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:47.920+0000 7f4fa6608440 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-16T12:42:48.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:48.446+0000 7f4fa6608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:49.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:49.210+0000 7f4fa6608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:49.979 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:49.978+0000 7f4fa6608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:50.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-16T12:42:50.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.887+0000 7fb23c651440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.887+0000 7fb23c651440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.887+0000 7fb23c651440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.887+0000 7fb23c651440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.887+0000 7fb23c651440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:42:50.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:50.888+0000 7fb23c651440 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-16T12:42:51.413 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:51.412+0000 7fb23c651440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:52.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:52.175+0000 7fb23c651440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:52.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:52.938+0000 7fb23c651440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:53.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-16T12:42:53.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.846+0000 7f35f4e0f440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.846+0000 7f35f4e0f440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.846+0000 7f35f4e0f440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.846+0000 7f35f4e0f440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.846+0000 7f35f4e0f440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:42:53.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:53.847+0000 7f35f4e0f440 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-16T12:42:54.377 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:54.375+0000 7f35f4e0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:55.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:55.142+0000 7f35f4e0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:55.905 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-16T12:42:55.904+0000 7f35f4e0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:42:56.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-16T12:42:56.730 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T12:42:56.731 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:56.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-16T12:42:56.760 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:56.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-16T12:42:56.799 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:56.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-16T12:42:56.866 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:56.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-16T12:42:56.932 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T12:42:56.932 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-16T12:42:56.951 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:56.951 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-16T12:42:56.988 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:56.988 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-16T12:42:57.044 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.044 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-16T12:42:57.099 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.099 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-16T12:42:57.155 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.156 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-16T12:42:57.211 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.212 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-16T12:42:57.267 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.267 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-16T12:42:57.322 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:57.322 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-16T12:42:57.377 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T12:42:57.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDEJ+hm/0QkLBAAX53MGU30zmfq9XbhCVM/FA== 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDHJ+hm3IczNhAAMDLZyP/xm3cmDdlyhiK0fA== 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDKJ+hmQ8npNBAAMgciotFWCZHU7MpbZgP2FA== 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDNJ+hm9l92MhAAWTDIyPHail/8aVdqAfOxog== 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-09-16T12:42:57.440 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDEJ+hmCzeDLhAAEujKksluT2QqZsHHj4NNiQ== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDEJ+hmDRp8MRAAQP1oBseYMe5e9QQD20DmuQ== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDEJ+hmvSjRNhAA0BhMioX07JiSGuuA22rLYw== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hmDiGQABAATLffbTHWrf/4cYGt7aARIQ== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hmjzreBRAADByqXai1pGktsXmvSYxwNw== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hm7KcxCxAA1U4X0I/oMEZ/jhZVaWCyPQ== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.4] 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hmRGaREBAAnxJFqosVDBgj/OEPs/VyGw== 2024-09-16T12:42:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.5] 2024-09-16T12:42:57.442 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hmV/fkFRAA9y6s3/HJK01h8Ad2AQArNQ== 2024-09-16T12:42:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.6] 2024-09-16T12:42:57.442 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFJ+hmMlE9GxAAC7P5eGcU7Tox7zZItQRVUQ== 2024-09-16T12:42:57.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T12:42:57.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:57.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:57.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:42:57.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:57.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:58.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:58.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:42:58.111 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T12:42:58.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-16T12:42:58.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-16T12:42:58.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-16T12:42:58.253 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:42:58.308 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-16T12:42:58.309 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-16T12:42:58.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-16T12:42:58.351 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-16T12:42:58.351 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-16T12:42:58.351 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-16T12:42:58.351 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-16T12:42:58.354 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-16T12:42:58.354 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T12:42:58.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-16T12:42:58.515 INFO:teuthology.orchestra.run.smithi191.stderr:adjusted tunables profile to default 2024-09-16T12:42:58.521 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:58.519+0000 7f9b39fc0ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T12:42:58.526 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-16T12:42:58.527 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:58.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-16T12:42:58.551 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:58.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-16T12:42:58.560 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:58.558+0000 7f9b39fc0ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T12:42:58.606 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:58.605+0000 7f9b39fc0ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T12:42:58.616 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:42:58.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-16T12:42:58.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 486ac1d9-0c28-4a08-9b32-aa19b63c78d8 0 2024-09-16T12:42:58.695 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:58.693+0000 7f9b39fc0ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:42:58.783 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:58.781+0000 7f9b39fc0ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T12:42:58.887 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-09-16T12:42:58.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c 1 2024-09-16T12:42:59.007 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:59.005+0000 7f9b39fc0ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T12:42:59.050 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:59.048+0000 7f9b39fc0ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T12:42:59.064 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-09-16T12:42:59.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 9894349c-b123-4be0-be82-1720ff2b1d0a 2 2024-09-16T12:42:59.148 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:59.146+0000 7f9b39fc0ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T12:42:59.239 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-09-16T12:42:59.250 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-16T12:42:59.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-16T12:42:59.252 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-16T12:42:59.253 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-16T12:42:59.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-16T12:42:59.255 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-16T12:42:59.255 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-16T12:42:59.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-16T12:42:59.257 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-16T12:42:59.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:42:59.420 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:42:59.420 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":5,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:42:59.237369+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:59.430 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-16T12:42:59.431 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T12:42:59.734 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:42:59.817 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:59.815+0000 7f9b39fc0ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T12:42:59.848 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T12:42:59.846+0000 7f966720f440 -1 Falling back to public interface 2024-09-16T12:42:59.853 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:42:59.851+0000 7fb63e51f440 -1 Falling back to public interface 2024-09-16T12:42:59.853 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T12:42:59.852+0000 7f09ede32440 -1 Falling back to public interface 2024-09-16T12:42:59.860 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:42:59.858+0000 7f9b39fc0ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T12:42:59.889 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-16T12:42:59.889 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":5,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:42:59.237369+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:42:59.899 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:43:00.167 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.165+0000 7f9b39fc0ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T12:43:00.245 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.243+0000 7f9b39fc0ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T12:43:00.332 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.330+0000 7f9b39fc0ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:43:00.372 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.371+0000 7f9b39fc0ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T12:43:00.463 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.461+0000 7f9b39fc0ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T12:43:00.512 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.510+0000 7f9b39fc0ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T12:43:00.552 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.550+0000 7f9b39fc0ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T12:43:00.644 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.642+0000 7f9b39fc0ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T12:43:00.689 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.687+0000 7f9b39fc0ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T12:43:00.735 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.733+0000 7f9b39fc0ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T12:43:00.786 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.783+0000 7f9b39fc0ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T12:43:00.826 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.825+0000 7f9b39fc0ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T12:43:00.866 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:00.864+0000 7f9b39fc0ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T12:43:01.403 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:01.401+0000 7f9b39fc0ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T12:43:02.411 INFO:tasks.ceph.cluster2.mgr.x.smithi177.stderr:2024-09-16T12:43:02.409+0000 7f9b1e530640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T12:43:03.331 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:43:03.330+0000 7fb63e51f440 -1 osd.2 0 log_to_monitors true 2024-09-16T12:43:03.342 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T12:43:03.340+0000 7f09ede32440 -1 osd.1 0 log_to_monitors true 2024-09-16T12:43:03.342 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T12:43:03.341+0000 7f966720f440 -1 osd.0 0 log_to_monitors true 2024-09-16T12:43:06.180 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:43:06.179+0000 7fb638ff6640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T12:43:06.186 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:43:06.185+0000 7fb631fd7640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:43:06.198 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T12:43:06.197+0000 7f09ea7f9640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T12:43:06.202 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:43:06.204 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T12:43:06.203+0000 7f96619f6640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T12:43:06.205 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T12:43:06.203+0000 7f09e1e84640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:43:06.209 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T12:43:06.208+0000 7f965a9e8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:43:06.453 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-16T12:43:06.453 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":7,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:43:04.409931+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:43:06.464 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:43:12.767 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:43:13.031 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-16T12:43:13.031 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":11,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:43:10.187616+0000","last_up_change":"2024-09-16T12:43:07.180410+0000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:43:07.438466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":33180},{"type":"v1","addr":"172.21.15.191:6801","nonce":33180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":33180},{"type":"v1","addr":"172.21.15.191:6803","nonce":33180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":33180},{"type":"v1","addr":"172.21.15.191:6807","nonce":33180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":33180},{"type":"v1","addr":"172.21.15.191:6805","nonce":33180}]},"public_addr":"172.21.15.191:6801/33180","cluster_addr":"172.21.15.191:6803/33180","heartbeat_back_addr":"172.21.15.191:6807/33180","heartbeat_front_addr":"172.21.15.191:6805/33180","state":["exists","up"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":33181},{"type":"v1","addr":"172.21.15.191:6812","nonce":33181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":33181},{"type":"v1","addr":"172.21.15.191:6816","nonce":33181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":33181},{"type":"v1","addr":"172.21.15.191:6823","nonce":33181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":33181},{"type":"v1","addr":"172.21.15.191:6820","nonce":33181}]},"public_addr":"172.21.15.191:6812/33181","cluster_addr":"172.21.15.191:6816/33181","heartbeat_back_addr":"172.21.15.191:6823/33181","heartbeat_front_addr":"172.21.15.191:6820/33181","state":["exists","up"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":33182},{"type":"v1","addr":"172.21.15.191:6809","nonce":33182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6811","nonce":33182},{"type":"v1","addr":"172.21.15.191:6813","nonce":33182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":33182},{"type":"v1","addr":"172.21.15.191:6821","nonce":33182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":33182},{"type":"v1","addr":"172.21.15.191:6817","nonce":33182}]},"public_addr":"172.21.15.191:6809/33182","cluster_addr":"172.21.15.191:6813/33182","heartbeat_back_addr":"172.21.15.191:6821/33182","heartbeat_front_addr":"172.21.15.191:6817/33182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.365008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.385578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.340888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:43:13.042 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T12:43:13.042 INFO:tasks.ceph:Creating RBD pool 2024-09-16T12:43:13.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-16T12:43:14.202 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'rbd' created 2024-09-16T12:43:14.219 DEBUG:teuthology.orchestra.run.smithi191:> rbd --cluster cluster2 pool init rbd 2024-09-16T12:43:17.222 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-16T12:43:17.223 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-16T12:43:17.223 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-16T12:43:17.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-16T12:43:17.223 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T12:43:17.540 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:17.551 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":37310},{"type":"v1","addr":"172.21.15.177:6827","nonce":37310}]},"active_addr":"172.21.15.177:6827/37310","active_change":"2024-09-16T12:43:01.403311+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":298088364}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1585026248}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3702855633}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":398027686}]}]} 2024-09-16T12:43:17.553 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-16T12:43:17.553 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-16T12:43:17.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:43:17.812 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:17.812 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":15,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:43:17.204810+0000","last_up_change":"2024-09-16T12:43:07.180410+0000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:43:07.438466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:43:13.318047+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":33180},{"type":"v1","addr":"172.21.15.191:6801","nonce":33180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":33180},{"type":"v1","addr":"172.21.15.191:6803","nonce":33180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":33180},{"type":"v1","addr":"172.21.15.191:6807","nonce":33180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":33180},{"type":"v1","addr":"172.21.15.191:6805","nonce":33180}]},"public_addr":"172.21.15.191:6801/33180","cluster_addr":"172.21.15.191:6803/33180","heartbeat_back_addr":"172.21.15.191:6807/33180","heartbeat_front_addr":"172.21.15.191:6805/33180","state":["exists","up"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":33181},{"type":"v1","addr":"172.21.15.191:6812","nonce":33181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":33181},{"type":"v1","addr":"172.21.15.191:6816","nonce":33181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":33181},{"type":"v1","addr":"172.21.15.191:6823","nonce":33181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":33181},{"type":"v1","addr":"172.21.15.191:6820","nonce":33181}]},"public_addr":"172.21.15.191:6812/33181","cluster_addr":"172.21.15.191:6816/33181","heartbeat_back_addr":"172.21.15.191:6823/33181","heartbeat_front_addr":"172.21.15.191:6820/33181","state":["exists","up"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":33182},{"type":"v1","addr":"172.21.15.191:6809","nonce":33182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6811","nonce":33182},{"type":"v1","addr":"172.21.15.191:6813","nonce":33182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":33182},{"type":"v1","addr":"172.21.15.191:6821","nonce":33182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":33182},{"type":"v1","addr":"172.21.15.191:6817","nonce":33182}]},"public_addr":"172.21.15.191:6809/33182","cluster_addr":"172.21.15.191:6813/33182","heartbeat_back_addr":"172.21.15.191:6821/33182","heartbeat_front_addr":"172.21.15.191:6817/33182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.365008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.385578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.340888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:43:17.825 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-16T12:43:17.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:43:18.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:18.086 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":15,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T12:43:17.204810+0000","last_up_change":"2024-09-16T12:43:07.180410+0000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:43:07.438466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:43:13.318047+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":33180},{"type":"v1","addr":"172.21.15.191:6801","nonce":33180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":33180},{"type":"v1","addr":"172.21.15.191:6803","nonce":33180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":33180},{"type":"v1","addr":"172.21.15.191:6807","nonce":33180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":33180},{"type":"v1","addr":"172.21.15.191:6805","nonce":33180}]},"public_addr":"172.21.15.191:6801/33180","cluster_addr":"172.21.15.191:6803/33180","heartbeat_back_addr":"172.21.15.191:6807/33180","heartbeat_front_addr":"172.21.15.191:6805/33180","state":["exists","up"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":33181},{"type":"v1","addr":"172.21.15.191:6812","nonce":33181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":33181},{"type":"v1","addr":"172.21.15.191:6816","nonce":33181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":33181},{"type":"v1","addr":"172.21.15.191:6823","nonce":33181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":33181},{"type":"v1","addr":"172.21.15.191:6820","nonce":33181}]},"public_addr":"172.21.15.191:6812/33181","cluster_addr":"172.21.15.191:6816/33181","heartbeat_back_addr":"172.21.15.191:6823/33181","heartbeat_front_addr":"172.21.15.191:6820/33181","state":["exists","up"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":33182},{"type":"v1","addr":"172.21.15.191:6809","nonce":33182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6811","nonce":33182},{"type":"v1","addr":"172.21.15.191:6813","nonce":33182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":33182},{"type":"v1","addr":"172.21.15.191:6821","nonce":33182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":33182},{"type":"v1","addr":"172.21.15.191:6817","nonce":33182}]},"public_addr":"172.21.15.191:6809/33182","cluster_addr":"172.21.15.191:6813/33182","heartbeat_back_addr":"172.21.15.191:6821/33182","heartbeat_front_addr":"172.21.15.191:6817/33182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.365008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.385578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.340888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:43:18.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-16T12:43:18.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-16T12:43:18.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-16T12:43:18.231 INFO:teuthology.orchestra.run.smithi191.stdout:34359738375 2024-09-16T12:43:18.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:43:18.247 INFO:teuthology.orchestra.run.smithi191.stdout:34359738375 2024-09-16T12:43:18.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:43:18.248 INFO:teuthology.orchestra.run.smithi191.stdout:34359738374 2024-09-16T12:43:18.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:43:18.530 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-16T12:43:18.542 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738373 for osd.2 2024-09-16T12:43:18.561 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-16T12:43:18.574 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738373 for osd.1 2024-09-16T12:43:18.585 INFO:teuthology.orchestra.run.smithi191.stdout:34359738372 2024-09-16T12:43:18.598 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-16T12:43:19.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:43:19.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:43:19.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:43:19.851 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-16T12:43:19.866 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738373 for osd.2 2024-09-16T12:43:19.888 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-16T12:43:19.900 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738373 for osd.1 2024-09-16T12:43:19.905 INFO:teuthology.orchestra.run.smithi191.stdout:34359738372 2024-09-16T12:43:19.917 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-16T12:43:20.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:43:20.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:43:20.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:43:21.159 INFO:teuthology.orchestra.run.smithi191.stdout:34359738375 2024-09-16T12:43:21.172 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738375 for osd.2 2024-09-16T12:43:21.172 DEBUG:teuthology.parallel:result is None 2024-09-16T12:43:21.197 INFO:teuthology.orchestra.run.smithi191.stdout:34359738375 2024-09-16T12:43:21.209 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738375 for osd.1 2024-09-16T12:43:21.209 DEBUG:teuthology.parallel:result is None 2024-09-16T12:43:21.234 INFO:teuthology.orchestra.run.smithi191.stdout:34359738374 2024-09-16T12:43:21.246 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-09-16T12:43:21.247 DEBUG:teuthology.parallel:result is None 2024-09-16T12:43:21.247 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T12:43:21.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T12:43:21.542 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:21.542 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T12:43:21.554 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T12:43:21.410294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893620,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221900,"statfs":{"total":289910292480,"available":288995225600,"internally_reserved":0,"allocated":1581056,"data_stored":1293777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.207540"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216670+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T12:43:17.216670+0000","last_peered":"2024-09-16T12:43:17.216670+0000","last_clean":"2024-09-16T12:43:17.216670+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T12:43:17.216670+0000","last_undegraded":"2024-09-16T12:43:17.216670+0000","last_fullsized":"2024-09-16T12:43:17.216670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.217059+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T12:43:17.217059+0000","last_peered":"2024-09-16T12:43:17.217059+0000","last_clean":"2024-09-16T12:43:17.217059+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T12:43:17.217059+0000","last_undegraded":"2024-09-16T12:43:17.217059+0000","last_fullsized":"2024-09-16T12:43:17.217059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216915+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T12:43:17.216915+0000","last_peered":"2024-09-16T12:43:17.216915+0000","last_clean":"2024-09-16T12:43:17.216915+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T12:43:17.216915+0000","last_undegraded":"2024-09-16T12:43:17.216915+0000","last_fullsized":"2024-09-16T12:43:17.216915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.217173+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T12:43:17.217173+0000","last_peered":"2024-09-16T12:43:17.217173+0000","last_clean":"2024-09-16T12:43:17.217173+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T12:43:17.217173+0000","last_undegraded":"2024-09-16T12:43:17.217173+0000","last_fullsized":"2024-09-16T12:43:17.217173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.214538+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T12:43:17.214538+0000","last_peered":"2024-09-16T12:43:17.214538+0000","last_clean":"2024-09-16T12:43:17.214538+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T12:43:17.214538+0000","last_undegraded":"2024-09-16T12:43:17.214538+0000","last_fullsized":"2024-09-16T12:43:17.214538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.345413+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T12:43:17.345413+0000","last_peered":"2024-09-16T12:43:17.345413+0000","last_clean":"2024-09-16T12:43:17.345413+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T12:43:17.345413+0000","last_undegraded":"2024-09-16T12:43:17.345413+0000","last_fullsized":"2024-09-16T12:43:17.345413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.345366+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T12:43:17.345366+0000","last_peered":"2024-09-16T12:43:17.345366+0000","last_clean":"2024-09-16T12:43:17.345366+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T12:43:17.345366+0000","last_undegraded":"2024-09-16T12:43:17.345366+0000","last_fullsized":"2024-09-16T12:43:17.345366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216614+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T12:43:17.216614+0000","last_peered":"2024-09-16T12:43:17.216614+0000","last_clean":"2024-09-16T12:43:17.216614+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T12:43:17.216614+0000","last_undegraded":"2024-09-16T12:43:17.216614+0000","last_fullsized":"2024-09-16T12:43:17.216614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":80,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T12:43:16.213358+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T12:43:16.213358+0000","last_peered":"2024-09-16T12:43:16.213358+0000","last_clean":"2024-09-16T12:43:16.213358+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T12:43:16.213358+0000","last_undegraded":"2024-09-16T12:43:16.213358+0000","last_fullsized":"2024-09-16T12:43:16.213358+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":126976,"data_stored":37414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298064,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073776,"statfs":{"total":96636764160,"available":96331546624,"internally_reserved":0,"allocated":724992,"data_stored":627801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:43:21.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T12:43:21.810 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:21.810 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T12:43:21.822 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T12:43:21.410294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893620,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221900,"statfs":{"total":289910292480,"available":288995225600,"internally_reserved":0,"allocated":1581056,"data_stored":1293777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.207540"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216670+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T12:43:17.216670+0000","last_peered":"2024-09-16T12:43:17.216670+0000","last_clean":"2024-09-16T12:43:17.216670+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T12:43:17.216670+0000","last_undegraded":"2024-09-16T12:43:17.216670+0000","last_fullsized":"2024-09-16T12:43:17.216670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.217059+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T12:43:17.217059+0000","last_peered":"2024-09-16T12:43:17.217059+0000","last_clean":"2024-09-16T12:43:17.217059+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T12:43:17.217059+0000","last_undegraded":"2024-09-16T12:43:17.217059+0000","last_fullsized":"2024-09-16T12:43:17.217059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216915+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T12:43:17.216915+0000","last_peered":"2024-09-16T12:43:17.216915+0000","last_clean":"2024-09-16T12:43:17.216915+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T12:43:17.216915+0000","last_undegraded":"2024-09-16T12:43:17.216915+0000","last_fullsized":"2024-09-16T12:43:17.216915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.217173+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T12:43:17.217173+0000","last_peered":"2024-09-16T12:43:17.217173+0000","last_clean":"2024-09-16T12:43:17.217173+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T12:43:17.217173+0000","last_undegraded":"2024-09-16T12:43:17.217173+0000","last_fullsized":"2024-09-16T12:43:17.217173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.214538+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T12:43:17.214538+0000","last_peered":"2024-09-16T12:43:17.214538+0000","last_clean":"2024-09-16T12:43:17.214538+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T12:43:17.214538+0000","last_undegraded":"2024-09-16T12:43:17.214538+0000","last_fullsized":"2024-09-16T12:43:17.214538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.345413+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T12:43:17.345413+0000","last_peered":"2024-09-16T12:43:17.345413+0000","last_clean":"2024-09-16T12:43:17.345413+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T12:43:17.345413+0000","last_undegraded":"2024-09-16T12:43:17.345413+0000","last_fullsized":"2024-09-16T12:43:17.345413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.345366+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T12:43:17.345366+0000","last_peered":"2024-09-16T12:43:17.345366+0000","last_clean":"2024-09-16T12:43:17.345366+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T12:43:17.345366+0000","last_undegraded":"2024-09-16T12:43:17.345366+0000","last_fullsized":"2024-09-16T12:43:17.345366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:43:17.216614+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T12:43:17.216614+0000","last_peered":"2024-09-16T12:43:17.216614+0000","last_clean":"2024-09-16T12:43:17.216614+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T12:43:17.216614+0000","last_undegraded":"2024-09-16T12:43:17.216614+0000","last_fullsized":"2024-09-16T12:43:17.216614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":80,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T12:43:16.213358+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T12:43:16.213358+0000","last_peered":"2024-09-16T12:43:16.213358+0000","last_clean":"2024-09-16T12:43:16.213358+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T12:43:16.213358+0000","last_undegraded":"2024-09-16T12:43:16.213358+0000","last_fullsized":"2024-09-16T12:43:16.213358+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":126976,"data_stored":37414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298064,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073776,"statfs":{"total":96636764160,"available":96331546624,"internally_reserved":0,"allocated":724992,"data_stored":627801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:43:21.823 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T12:43:21.823 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-16T12:43:21.823 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-16T12:43:21.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-16T12:43:22.113 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T12:43:22.114 INFO:teuthology.orchestra.run.smithi191.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T12:43:22.126 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-16T12:43:22.126 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T12:43:22.136 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T12:43:22.136 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T12:43:22.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:22.466 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror 2024-09-16T12:43:22.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:22.806 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.0 2024-09-16T12:43:22.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:23.145 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.1 2024-09-16T12:43:23.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:23.494 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.2 2024-09-16T12:43:23.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:23.834 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.3 2024-09-16T12:43:23.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:24.173 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror 2024-09-16T12:43:24.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:24.517 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.0 2024-09-16T12:43:24.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:24.857 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.1 2024-09-16T12:43:24.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.2 2024-09-16T12:43:25.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:43:25.547 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.3 2024-09-16T12:43:25.561 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T12:43:25.572 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T12:43:25.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T12:43:25.575 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T12:43:25.611 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T12:43:25.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T12:43:25.614 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T12:43:25.622 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T12:43:25.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T12:43:25.624 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T12:43:25.633 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T12:43:25.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T12:43:25.635 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T12:43:25.644 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T12:43:25.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T12:43:25.646 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T12:43:25.658 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T12:43:25.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T12:43:25.660 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T12:43:25.670 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T12:43:25.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T12:43:25.673 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T12:43:25.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T12:43:25.681 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T12:43:25.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T12:43:25.684 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T12:43:25.695 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726490605 2024-09-16T12:43:25.695 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T12:43:25.695 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T12:43:25.704 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726490605 2024-09-16T12:43:25.704 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T12:43:25.704 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T12:43:25.716 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:43:25.717 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T12:43:25.717 DEBUG:teuthology.orchestra.run.smithi191:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:43:25.717 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-16T12:43:25.717 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-16T12:43:25.718 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-16T12:43:25.718 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-16T12:43:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:43:25.734 INFO:teuthology.orchestra.run.smithi191.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-16T12:43:25.734 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:43:25.789 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:43:25.789 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-16T12:43:25.844 INFO:tasks.workunit:timeout=6h 2024-09-16T12:43:25.844 INFO:tasks.workunit:cleanup=True 2024-09-16T12:43:25.845 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:43:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-16T12:44:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-16T12:44:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T12:44:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: git switch -c 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Or undo this operation with: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: git switch - 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:44:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:44:15.884 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-16T12:44:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T12:44:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T12:44:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T12:44:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T12:44:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T12:44:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T12:44:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T12:44:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T12:44:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T12:44:16.888 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-16T12:44:16.888 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-16T12:44:16.944 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-16T12:44:16.945 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-16T12:44:16.945 DEBUG:teuthology.orchestra.run.smithi191:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T12:44:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ IMAGE_COUNT=50 2024-09-16T12:44:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ export LOCKDEP=0 2024-09-16T12:44:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ LOCKDEP=0 2024-09-16T12:44:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ type xmlstarlet 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ XMLSTARLET=xmlstarlet 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CLUSTER1=cluster1 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CLUSTER2=cluster2 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-16T12:44:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ POOL=mirror 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ PARENT_POOL=mirror_parent 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ NS1=ns1 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ NS2=ns2 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ TEMPDIR= 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ID=mirror 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_POOL_MODE=image 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS='--id mirror' 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T12:44:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-16T12:44:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-16T12:44:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_BIN=. 2024-09-16T12:44:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_SRC=. 2024-09-16T12:44:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-16T12:44:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 -gt 0 ']' 2024-09-16T12:44:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup 2024-09-16T12:44:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local c 2024-09-16T12:44:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-16T12:44:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_tempdir 2024-09-16T12:44:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mktemp -d 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ TEMPDIR=/tmp/tmp.cccYcKFUz2 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z 1 ']' 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_pools cluster1 cluster2 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_map_file 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_addr 2024-09-16T12:44:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local admin_key_file 2024-09-16T12:44:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid 2024-09-16T12:44:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-16T12:44:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' already exists 2024-09-16T12:44:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-16T12:44:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' already exists 2024-09-16T12:44:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-16T12:44:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-16T12:44:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-16T12:44:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-16T12:44:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-16T12:44:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-16T12:44:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-16T12:44:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-16T12:44:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-16T12:44:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-16T12:44:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster1 mirror cluster2 2024-09-16T12:44:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster2 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:44:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-16T12:44:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=26908201-7ad5-4cae-90e4-65f0c24b4781 2024-09-16T12:44:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T12:44:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror_parent 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster2 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-16T12:44:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-16T12:44:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-16T12:44:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:44:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:44:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-16T12:44:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=474dda11-77c2-4fc9-b5b4-a0f341b7f4fd 2024-09-16T12:44:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T12:44:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:44:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_pools cluster2 cluster1 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_map_file 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_addr 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local admin_key_file 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-16T12:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' already exists 2024-09-16T12:44:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-16T12:44:26.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T12:44:26.007+0000 7f5b43b215c0 -1 rbd::mirror::PoolReplayer: 0x561ba555d500 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:26.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T12:44:26.009+0000 7f4cbad2d5c0 -1 rbd::mirror::PoolReplayer: 0x55e53b01f500 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:26.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:44:26.007+0000 7f909e1235c0 -1 rbd::mirror::PoolReplayer: 0x559dd76f9500 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:26.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T12:44:26.014+0000 7f8ae93305c0 -1 rbd::mirror::PoolReplayer: 0x55d9e2d91500 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' already exists 2024-09-16T12:44:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-16T12:44:26.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:44:26.995+0000 7f909e1235c0 -1 rbd::mirror::PoolReplayer: 0x559dd8de6f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:26.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T12:44:26.997+0000 7f5b43b215c0 -1 rbd::mirror::PoolReplayer: 0x561ba6c6af00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:27.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T12:44:26.998+0000 7f8ae93305c0 -1 rbd::mirror::PoolReplayer: 0x55d9e449af00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:27.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T12:44:27.005+0000 7f4cbad2d5c0 -1 rbd::mirror::PoolReplayer: 0x55e53c71cf00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T12:44:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-16T12:44:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T12:44:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-16T12:44:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-16T12:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-16T12:44:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-16T12:44:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-16T12:44:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster2 mirror cluster1 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster1 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:44:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-16T12:44:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=14122d5b-3729-428d-a1df-d1465a5dea57 2024-09-16T12:44:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T12:44:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror_parent 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster1 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:44:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-16T12:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=f359323e-4c0f-4d19-bae5-71def3c788eb 2024-09-16T12:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T12:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirrors cluster1 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:0 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:0 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:0 2024-09-16T12:44:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:1 2024-09-16T12:44:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:2 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:2 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:2 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=2 2024-09-16T12:44:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=2 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:3 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:3 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:3 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=3 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=3 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirrors cluster2 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:0 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:0 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:0 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-16T12:44:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:1 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:1 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:1 2024-09-16T12:44:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=1 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=1 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:2 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:2 2024-09-16T12:44:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:3 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:3 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:3 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=3 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-16T12:44:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=3 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 12:44:32 TEST: add image and test replay after client crashes 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 12:44:32 TEST: add image and test replay after client crashes 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=test 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror test 512M 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T12:44:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T12:44:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:44:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-09-16T12:44:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:44:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-16T12:44:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:44:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10b92b8d539c 2024-09-16T12:44:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10b92b8d539c 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:32 2024 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:44:32 2024 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:32 2024 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:44:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:44:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:44:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:44:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:44:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:44:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:44:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:44:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:44:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:44:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:34.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 1 2024-09-16T12:44:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 2 2024-09-16T12:44:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 4 2024-09-16T12:44:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 8 2024-09-16T12:44:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:44:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 8 2024-09-16T12:44:56.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:44:56.005+0000 7f909e1235c0 -1 rbd::mirror::Mirror: 0x559dd68df440 update_pool_replayers: restarting failed pool replayer for uuid: 26908201-7ad5-4cae-90e4-65f0c24b4781 cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T12:44:56.006+0000 7f5b43b215c0 -1 rbd::mirror::Mirror: 0x561ba4763440 update_pool_replayers: restarting failed pool replayer for uuid: 26908201-7ad5-4cae-90e4-65f0c24b4781 cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T12:44:56.006+0000 7f8ae93305c0 -1 rbd::mirror::Mirror: 0x55d9e1f97440 update_pool_replayers: restarting failed pool replayer for uuid: 26908201-7ad5-4cae-90e4-65f0c24b4781 cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T12:44:56.013+0000 7f4cbad2d5c0 -1 rbd::mirror::Mirror: 0x55e53a219440 update_pool_replayers: restarting failed pool replayer for uuid: 26908201-7ad5-4cae-90e4-65f0c24b4781 cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T12:44:56.094+0000 7f8ae93305c0 -1 rbd::mirror::Mirror: 0x55d9e1f97440 update_pool_replayers: restarting failed pool replayer for uuid: 474dda11-77c2-4fc9-b5b4-a0f341b7f4fd cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:44:56.110+0000 7f909e1235c0 -1 rbd::mirror::Mirror: 0x559dd68df440 update_pool_replayers: restarting failed pool replayer for uuid: 474dda11-77c2-4fc9-b5b4-a0f341b7f4fd cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T12:44:56.111+0000 7f5b43b215c0 -1 rbd::mirror::Mirror: 0x561ba4763440 update_pool_replayers: restarting failed pool replayer for uuid: 474dda11-77c2-4fc9-b5b4-a0f341b7f4fd cluster: cluster2 client: client.mirror 2024-09-16T12:44:56.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T12:44:56.120+0000 7f4cbad2d5c0 -1 rbd::mirror::Mirror: 0x55e53a219440 update_pool_replayers: restarting failed pool replayer for uuid: 474dda11-77c2-4fc9-b5b4-a0f341b7f4fd cluster: cluster2 client: client.mirror 2024-09-16T12:44:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:44:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:44:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:44:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:44:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:44:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:44:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:44:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:44:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:44:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:44:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:44:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:44:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:44:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:44:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:44:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:44:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:44:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:44:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:44:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:44:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:44:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:44:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:44:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:44:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:44:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:44:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:44:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:44:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:44:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:44:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:44:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:44:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:44:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:44:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:44:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name= 2024-09-16T12:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:44:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:44:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T12:44:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:44:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:44:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 29514752 bytes 2024-09-16T12:44:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 50643968 bytes 2024-09-16T12:44:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 74960896 bytes 2024-09-16T12:44:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 133409792 bytes 2024-09-16T12:44:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 200706048 bytes 2024-09-16T12:45:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 265778176 bytes 2024-09-16T12:45:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 304530432 bytes 2024-09-16T12:45:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 338035712 bytes 2024-09-16T12:45:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 364255232 bytes 2024-09-16T12:45:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 366448640 bytes 2024-09-16T12:45:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 407275520 bytes 2024-09-16T12:45:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 407882752 bytes 2024-09-16T12:45:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 413521920 bytes 2024-09-16T12:45:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 463860736 bytes 2024-09-16T12:45:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 508228608 bytes 2024-09-16T12:45:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 508837888 bytes 2024-09-16T12:45:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 528498688 bytes 2024-09-16T12:45:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 609910784 bytes 2024-09-16T12:45:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 632839168 bytes 2024-09-16T12:45:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 705264640 bytes 2024-09-16T12:45:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 721513472 bytes 2024-09-16T12:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 739098624 bytes 2024-09-16T12:45:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 814653440 bytes 2024-09-16T12:45:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 841070592 bytes 2024-09-16T12:45:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 895876096 bytes 2024-09-16T12:45:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 909926400 bytes 2024-09-16T12:45:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 947900416 bytes 2024-09-16T12:45:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1059424256 bytes 2024-09-16T12:45:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1118587904 bytes 2024-09-16T12:45:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1203308544 bytes 2024-09-16T12:45:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1203881984 bytes 2024-09-16T12:45:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1234505728 bytes 2024-09-16T12:45:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1320992768 bytes 2024-09-16T12:45:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1343420416 bytes 2024-09-16T12:45:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1414203392 bytes 2024-09-16T12:45:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1429013504 bytes 2024-09-16T12:45:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1510811648 bytes 2024-09-16T12:45:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1544836096 bytes 2024-09-16T12:45:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1570545664 bytes 2024-09-16T12:45:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1595835392 bytes 2024-09-16T12:45:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1689819136 bytes 2024-09-16T12:45:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1760587776 bytes 2024-09-16T12:45:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1800745984 bytes 2024-09-16T12:45:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1828502528 bytes 2024-09-16T12:45:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1879429120 bytes 2024-09-16T12:45:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1960948736 bytes 2024-09-16T12:45:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2096259072 bytes 2024-09-16T12:45:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2113838080 bytes 2024-09-16T12:45:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2182247424 bytes 2024-09-16T12:45:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2266524672 bytes 2024-09-16T12:45:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2296700928 bytes 2024-09-16T12:45:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2303891456 bytes 2024-09-16T12:45:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2323916800 bytes 2024-09-16T12:45:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2364009472 bytes 2024-09-16T12:45:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2413886464 bytes 2024-09-16T12:45:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2476197888 bytes 2024-09-16T12:45:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2512780288 bytes 2024-09-16T12:45:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2522518528 bytes 2024-09-16T12:45:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2550545408 bytes 2024-09-16T12:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2551141376 bytes 2024-09-16T12:45:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2560281600 bytes 2024-09-16T12:45:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2583547904 bytes 2024-09-16T12:45:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2599325696 bytes 2024-09-16T12:45:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2655886336 bytes 2024-09-16T12:45:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2689784832 bytes 2024-09-16T12:45:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2749740032 bytes 2024-09-16T12:45:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2789685248 bytes 2024-09-16T12:45:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2838226944 bytes 2024-09-16T12:45:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2871392256 bytes 2024-09-16T12:45:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2896429056 bytes 2024-09-16T12:45:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2900369408 bytes 2024-09-16T12:45:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2949154816 bytes 2024-09-16T12:45:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2995479552 bytes 2024-09-16T12:45:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3028325376 bytes 2024-09-16T12:45:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3062232064 bytes 2024-09-16T12:45:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3081178112 bytes 2024-09-16T12:45:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3140600832 bytes 2024-09-16T12:45:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3143282688 bytes 2024-09-16T12:45:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3197774848 bytes 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:45:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-16T12:45:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:45:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-16T12:45:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-16T12:45:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:45:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:45:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:45:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-16T12:45:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:45:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-16T12:45:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 4 2024-09-16T12:45:16.718 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5339510497968154) > weight (0.25) 2024-09-16T12:45:16.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.975127991820888) > weight (0.25) 2024-09-16T12:45:16.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5464815681973574) > weight (0.25) 2024-09-16T12:45:16.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6818118040522023) > weight (0.25) 2024-09-16T12:45:16.719 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-16T12:45:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:45:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:45:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 8 2024-09-16T12:45:18.718 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9884986325473777) > weight (0.25) 2024-09-16T12:45:18.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.25814634023699146) > weight (0.25) 2024-09-16T12:45:18.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6123703737745049) > weight (0.25) 2024-09-16T12:45:18.719 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T12:45:18.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T12:45:18.720 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T12:45:18.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T12:45:18.718+0000 7f609abf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38222) UID: 1000 2024-09-16T12:45:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:45:19.720 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T12:45:19.721 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:45:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:45:19.721 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T12:45:19.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T12:45:19.722 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T12:45:19.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T12:45:19.725 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-16T12:45:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:45:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490673,"remote_snapshot_timestamp":1726490673,"replay_state":"idle"} 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:45:26 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:45:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:45:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:45:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490673,"remote_snapshot_timestamp":1726490673,"replay_state":"idle"} 2024-09-16T12:45:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:45:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:45:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:45:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:45:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:45:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:45:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:45:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:45:26' 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:45:26' 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:45:26' 2024-09-16T12:45:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:45:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='WARNINGOKWARNING142184299mirror.0smithi19117.2.7-1643-gd75f2382falseOK42224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490673,"remote_snapshot_timestamp":1726490673,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:45:26cluster268392c8a-786e-4db9-983c-eb2f365ca1cadown+unknownstatus not found' 2024-09-16T12:45:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:45:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:45:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:45:26' 2024-09-16T12:45:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:45:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:45:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:45:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:45:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:45:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap1 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-16T12:45:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:45:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:45:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:45:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:45:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:45:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:45:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:45:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:45:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:45:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:45:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:45:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:45:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:45:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:45:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:45:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:45:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:45:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:45:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:45:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:45:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:45:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:45:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:45:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:45:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:45:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:45:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:45:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:45:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:45:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:45:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:45:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:45:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:45:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:45:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:45:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:45:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:45:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:45:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:45:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:45:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:45:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:45:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:45:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:45:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:45:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:45:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:45:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:45:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:45:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:45:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:45:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:45:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:45:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:45:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:45:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 6 2024-09-16T12:45:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:45:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:45:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:45:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:45:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:45:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=6 2024-09-16T12:45:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:45:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:45:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:45:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:45:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:45:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:45:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=4 2024-09-16T12:45:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = 4 2024-09-16T12:45:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:45:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:45:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:45:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:45:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=4 2024-09-16T12:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = 4 2024-09-16T12:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:45:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:45:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = '' 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:45:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:45:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:45:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:45:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=6 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = 6 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-16T12:45:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:45:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:45:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:45:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-16T12:45:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:45:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:45:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:45:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T12:45:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:45:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:45:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:45:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:45:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:45:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:45:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:45:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:45:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:45:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:45:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap1-clean 2024-09-16T12:45:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-16T12:45:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-16T12:45:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:45:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:45:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:45:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:45:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:45:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:45:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:45:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 32495616 bytes 2024-09-16T12:45:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 53174272 bytes 2024-09-16T12:45:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 92140544 bytes 2024-09-16T12:45:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 138569728 bytes 2024-09-16T12:45:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 213564416 bytes 2024-09-16T12:45:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 237741056 bytes 2024-09-16T12:45:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 272830464 bytes 2024-09-16T12:45:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 288321536 bytes 2024-09-16T12:45:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 352433152 bytes 2024-09-16T12:45:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 366386176 bytes 2024-09-16T12:45:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 476342272 bytes 2024-09-16T12:45:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 504788992 bytes 2024-09-16T12:45:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 572743680 bytes 2024-09-16T12:45:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 646130688 bytes 2024-09-16T12:45:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 694832128 bytes 2024-09-16T12:45:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 734070784 bytes 2024-09-16T12:45:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 810906624 bytes 2024-09-16T12:45:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 865136640 bytes 2024-09-16T12:45:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 934489088 bytes 2024-09-16T12:45:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1027206144 bytes 2024-09-16T12:45:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1083964416 bytes 2024-09-16T12:45:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1118703616 bytes 2024-09-16T12:45:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1149260800 bytes 2024-09-16T12:45:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1170205696 bytes 2024-09-16T12:45:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1246942208 bytes 2024-09-16T12:45:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1281107968 bytes 2024-09-16T12:45:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1325655040 bytes 2024-09-16T12:45:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1457291264 bytes 2024-09-16T12:45:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1463029760 bytes 2024-09-16T12:45:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1542020096 bytes 2024-09-16T12:45:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1569531904 bytes 2024-09-16T12:45:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1586750464 bytes 2024-09-16T12:45:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1609040896 bytes 2024-09-16T12:45:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1684060160 bytes 2024-09-16T12:45:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1754811392 bytes 2024-09-16T12:45:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1755421696 bytes 2024-09-16T12:45:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1811601408 bytes 2024-09-16T12:45:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1839049728 bytes 2024-09-16T12:45:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1886275584 bytes 2024-09-16T12:45:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1941957632 bytes 2024-09-16T12:45:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1992853504 bytes 2024-09-16T12:45:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2046357504 bytes 2024-09-16T12:45:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:45:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:45:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:45:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:45:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:45:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:45:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17266756.27,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"} 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:45:56 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:45:56' 2024-09-16T12:45:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:45:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:45:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:45:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17266756.27,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"} 2024-09-16T12:45:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:45:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:45:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:45:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:45:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:45:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:45:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:45:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:45:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:45:56' 2024-09-16T12:45:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:45:56' 2024-09-16T12:45:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:45:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:45:56' 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:45:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:45:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142184299mirror.0smithi19117.2.7-1643-gd75f2382falseOK42224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":17266756.27,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:45:56cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:45:56' 2024-09-16T12:45:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:45:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:45:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:45:56' 2024-09-16T12:45:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:45:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:45:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:45:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:45:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:45:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:45:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-16T12:45:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap2 2024-09-16T12:45:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:45:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:45:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:45:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-16T12:45:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:46:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:46:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:46:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:46:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:46:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:46:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:46:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:46:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:46:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:46:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:46:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:46:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:46:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:46:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:46:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:46:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:46:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:46:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:46:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:46:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:46:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:46:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:46:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:46:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:46:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:46:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:46:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:46:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:46:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:46:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:46:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:46:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:46:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:46:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:46:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:46:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:46:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:46:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:46:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:46:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:46:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:46:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:46:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:46:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:46:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:46:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:46:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:46:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:46:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:46:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:46:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:46:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:46:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:46:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:46:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:46:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:46:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:46:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:46:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:46:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 9 2024-09-16T12:46:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:46:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:46:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:46:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:46:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:46:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=9 2024-09-16T12:46:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:46:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:46:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:46:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:46:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 9 = '' 2024-09-16T12:46:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:46:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:46:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:46:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:46:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:46:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:46:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:46:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:46:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 9 = '' 2024-09-16T12:46:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:46:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:46:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:46:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:46:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:46:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:46:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:46:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:46:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:46:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 9 = '' 2024-09-16T12:46:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:46:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:46:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:46:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=9 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 9 = 9 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:46:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:46:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 4 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:46:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:46:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:46:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap1-clean ']' 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:46:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:46:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:46:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:46:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:46:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:46:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:46:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap2-clean 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-16T12:46:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:46:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:46:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:46:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:46:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:46:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:46:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:46:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 66667520 bytes 2024-09-16T12:46:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 114496512 bytes 2024-09-16T12:46:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 156289024 bytes 2024-09-16T12:46:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 209546240 bytes 2024-09-16T12:46:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 266468352 bytes 2024-09-16T12:46:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 302654464 bytes 2024-09-16T12:46:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 306674688 bytes 2024-09-16T12:46:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 407503872 bytes 2024-09-16T12:46:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 514038784 bytes 2024-09-16T12:46:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 532329472 bytes 2024-09-16T12:46:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 532925440 bytes 2024-09-16T12:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 579376128 bytes 2024-09-16T12:46:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 579995648 bytes 2024-09-16T12:46:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 642134016 bytes 2024-09-16T12:46:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 688234496 bytes 2024-09-16T12:46:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 700321792 bytes 2024-09-16T12:46:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 754131968 bytes 2024-09-16T12:46:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 754758656 bytes 2024-09-16T12:46:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 777067520 bytes 2024-09-16T12:46:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 790025216 bytes 2024-09-16T12:46:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 851056640 bytes 2024-09-16T12:46:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 858050560 bytes 2024-09-16T12:46:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 874456064 bytes 2024-09-16T12:46:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 904994816 bytes 2024-09-16T12:46:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 993113088 bytes 2024-09-16T12:46:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1024589824 bytes 2024-09-16T12:46:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1094034432 bytes 2024-09-16T12:46:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1141541888 bytes 2024-09-16T12:46:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1210525696 bytes 2024-09-16T12:46:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1249099776 bytes 2024-09-16T12:46:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1324514304 bytes 2024-09-16T12:46:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1359323136 bytes 2024-09-16T12:46:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1367853056 bytes 2024-09-16T12:46:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1413593088 bytes 2024-09-16T12:46:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1435362304 bytes 2024-09-16T12:46:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1490701312 bytes 2024-09-16T12:46:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1498979328 bytes 2024-09-16T12:46:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1519294464 bytes 2024-09-16T12:46:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1564773376 bytes 2024-09-16T12:46:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1565380608 bytes 2024-09-16T12:46:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1571422208 bytes 2024-09-16T12:46:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1634556928 bytes 2024-09-16T12:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1641722880 bytes 2024-09-16T12:46:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1681565696 bytes 2024-09-16T12:46:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1725921280 bytes 2024-09-16T12:46:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1776294912 bytes 2024-09-16T12:46:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1776942080 bytes 2024-09-16T12:46:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:46:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"} 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:46:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:46:26 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:46:26' 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:46:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:46:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:46:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"} 2024-09-16T12:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:46:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:46:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:46:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:46:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:46:26' 2024-09-16T12:46:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:46:26' 2024-09-16T12:46:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:46:26' 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:46:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142184299mirror.0smithi19117.2.7-1643-gd75f2382falseOK42224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":259001344.0,"last_snapshot_bytes":518002688,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490742,"remote_snapshot_timestamp":1726490742,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:46:26cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:46:26' 2024-09-16T12:46:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:46:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:46:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:46:26' 2024-09-16T12:46:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:46:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:46:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:46:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:46:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap3 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-16T12:46:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:46:55.719 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T12:46:55.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T12:46:55.720 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T12:46:55.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T12:46:55.719+0000 7f5b3fdf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38070) UID: 1000 2024-09-16T12:46:55.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:47:02.722 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T12:47:02.722 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:47:02.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:47:02.722 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T12:47:02.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T12:47:02.723 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T12:47:02.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T12:47:02.726 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-16T12:47:05.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.283446838266059) > weight (0.25) 2024-09-16T12:47:05.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9275729366413895) > weight (0.25) 2024-09-16T12:47:05.728 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T12:47:05.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T12:47:05.728 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T12:47:05.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T12:47:05.727+0000 7ff5c45f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38193) UID: 1000 2024-09-16T12:47:05.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:47:09.729 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T12:47:09.729 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:47:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:47:09.729 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T12:47:09.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T12:47:09.730 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T12:47:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T12:47:09.733 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-16T12:47:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:47:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:47:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:47:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:47:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:47:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:47:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:47:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:47:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:47:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:47:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:47:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:47:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:47:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:47:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:47:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:47:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:47:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:47:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:47:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:47:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:47:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:47:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:47:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:47:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:47:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:47:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:47:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:47:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:47:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:47:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:47:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:47:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:47:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:47:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:47:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:47:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:47:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:47:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:47:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:47:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:47:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:47:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:47:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:47:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:47:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:47:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:47:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:47:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:47:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:47:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:47:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:47:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:47:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:47:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:47:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:47:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:47:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 13 2024-09-16T12:47:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:47:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:47:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:47:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:47:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:47:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=13 2024-09-16T12:47:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:47:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:47:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:47:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:47:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:47:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:47:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:47:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:47:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:47:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 13 = '' 2024-09-16T12:47:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:47:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:47:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:47:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:47:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 13 = '' 2024-09-16T12:47:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:47:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:47:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:47:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 13 = '' 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:47:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:47:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:47:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:47:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=13 2024-09-16T12:47:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 13 = 13 2024-09-16T12:47:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:47:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:47:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 6 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:46:33 2024 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:47:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap2-clean ']' 2024-09-16T12:47:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:47:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:47:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:47:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:47:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:47:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:47:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap3-clean 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-16T12:47:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:47:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T12:47:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:47:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:47:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 63951872 bytes 2024-09-16T12:47:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 107902976 bytes 2024-09-16T12:47:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 178300928 bytes 2024-09-16T12:47:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 200329216 bytes 2024-09-16T12:47:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 235345920 bytes 2024-09-16T12:47:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 261566464 bytes 2024-09-16T12:47:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 262180864 bytes 2024-09-16T12:47:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 288996352 bytes 2024-09-16T12:47:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 317563904 bytes 2024-09-16T12:47:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 318193664 bytes 2024-09-16T12:47:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 362142720 bytes 2024-09-16T12:47:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 391306240 bytes 2024-09-16T12:47:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 486232064 bytes 2024-09-16T12:47:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 520561664 bytes 2024-09-16T12:47:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 544893952 bytes 2024-09-16T12:47:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 558609408 bytes 2024-09-16T12:47:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 565460992 bytes 2024-09-16T12:47:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 659097600 bytes 2024-09-16T12:47:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 672004096 bytes 2024-09-16T12:47:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 768434176 bytes 2024-09-16T12:47:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 795797504 bytes 2024-09-16T12:47:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 809398272 bytes 2024-09-16T12:47:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 877393920 bytes 2024-09-16T12:47:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 944435200 bytes 2024-09-16T12:47:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 949244928 bytes 2024-09-16T12:47:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 987982848 bytes 2024-09-16T12:47:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1010401280 bytes 2024-09-16T12:47:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1041882112 bytes 2024-09-16T12:47:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1067097088 bytes 2024-09-16T12:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1079290880 bytes 2024-09-16T12:47:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1152158720 bytes 2024-09-16T12:47:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1172984832 bytes 2024-09-16T12:47:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1245008896 bytes 2024-09-16T12:47:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1275288576 bytes 2024-09-16T12:47:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1310117888 bytes 2024-09-16T12:47:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1321022464 bytes 2024-09-16T12:47:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1379089408 bytes 2024-09-16T12:47:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1461980160 bytes 2024-09-16T12:47:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1514389504 bytes 2024-09-16T12:47:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1556426752 bytes 2024-09-16T12:47:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1600707584 bytes 2024-09-16T12:47:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1646608384 bytes 2024-09-16T12:47:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1671094272 bytes 2024-09-16T12:47:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1725936640 bytes 2024-09-16T12:47:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1732586496 bytes 2024-09-16T12:47:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1786713088 bytes 2024-09-16T12:47:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1834953728 bytes 2024-09-16T12:47:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1845501952 bytes 2024-09-16T12:47:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1869967360 bytes 2024-09-16T12:47:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1918144512 bytes 2024-09-16T12:47:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2012550144 bytes 2024-09-16T12:47:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2069506048 bytes 2024-09-16T12:47:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2122019840 bytes 2024-09-16T12:47:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2149101568 bytes 2024-09-16T12:47:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2188367872 bytes 2024-09-16T12:47:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2199240704 bytes 2024-09-16T12:47:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2258016256 bytes 2024-09-16T12:47:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2338461696 bytes 2024-09-16T12:47:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2426364928 bytes 2024-09-16T12:47:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2501966848 bytes 2024-09-16T12:47:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2539872256 bytes 2024-09-16T12:47:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2624674816 bytes 2024-09-16T12:47:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2703022080 bytes 2024-09-16T12:47:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2728913920 bytes 2024-09-16T12:47:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2757970944 bytes 2024-09-16T12:47:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2798669824 bytes 2024-09-16T12:47:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2838822912 bytes 2024-09-16T12:47:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2892473344 bytes 2024-09-16T12:47:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2981264384 bytes 2024-09-16T12:47:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3022143488 bytes 2024-09-16T12:47:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3093011456 bytes 2024-09-16T12:47:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3124294656 bytes 2024-09-16T12:47:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3151879168 bytes 2024-09-16T12:47:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3197058048 bytes 2024-09-16T12:47:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3208453120 bytes 2024-09-16T12:47:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3231315968 bytes 2024-09-16T12:47:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3298503680 bytes 2024-09-16T12:47:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3353810944 bytes 2024-09-16T12:47:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3358392320 bytes 2024-09-16T12:47:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3366886400 bytes 2024-09-16T12:47:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3451648000 bytes 2024-09-16T12:47:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3533699072 bytes 2024-09-16T12:47:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:47:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":16686148.27,"bytes_per_snapshot":517192704.0,"last_snapshot_bytes":500584448,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490835,"remote_snapshot_timestamp":1726490835,"replay_state":"idle"} 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:47:26 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:47:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:47:26' 2024-09-16T12:47:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:47:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:47:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:47:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":16686148.27,"bytes_per_snapshot":517192704.0,"last_snapshot_bytes":500584448,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490835,"remote_snapshot_timestamp":1726490835,"replay_state":"idle"} 2024-09-16T12:47:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:47:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:47:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:47:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:47:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:47:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:47:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:47:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:47:26' 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:47:26' 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:47:26' 2024-09-16T12:47:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:47:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:47:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:47:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":16686148.27,"bytes_per_snapshot":517192704.0,"last_snapshot_bytes":500584448,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490835,"remote_snapshot_timestamp":1726490835,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:47:26cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:47:26' 2024-09-16T12:47:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:47:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:47:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:47:26' 2024-09-16T12:47:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:47:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:47:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:47:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:47:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:47:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:47:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-16T12:47:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap4 2024-09-16T12:47:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:47:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:47:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:47:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-16T12:47:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:48:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:48:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:48:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:48:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:48:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:48:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:48:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:48:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:48:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:48:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:48:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:48:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:48:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:48:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:48:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:48:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:48:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:48:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:48:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:48:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:48:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:48:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:48:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:48:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:48:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:48:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:48:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:48:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:48:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:48:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:48:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:48:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:48:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:48:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:48:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:48:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:48:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:48:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:48:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:48:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:48:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:48:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:48:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:48:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:48:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:48:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:48:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:48:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:48:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:48:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 17 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:48:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:48:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:48:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=17 2024-09-16T12:48:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:48:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:48:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:48:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:48:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:48:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:48:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:48:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-16T12:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:48:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:48:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:48:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:48:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:48:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:48:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:48:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:48:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:48:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-16T12:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:48:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:48:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:48:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:48:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:48:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=17 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = 17 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-16T12:48:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:48:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:48:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:48:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-16T12:48:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:48:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:48:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 8 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:46:33 2024 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:48:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:48:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:48:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap3-clean ']' 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:48:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:48:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:48:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:48:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:48:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:48:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:48:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap4-clean 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-16T12:48:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:48:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:48:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:48:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:48:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T12:48:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:48:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:48:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 104803328 bytes 2024-09-16T12:48:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 141750272 bytes 2024-09-16T12:48:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 183960576 bytes 2024-09-16T12:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 205629440 bytes 2024-09-16T12:48:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 244243456 bytes 2024-09-16T12:48:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 266700800 bytes 2024-09-16T12:48:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 319467520 bytes 2024-09-16T12:48:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 361469952 bytes 2024-09-16T12:48:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 383878144 bytes 2024-09-16T12:48:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 408095744 bytes 2024-09-16T12:48:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 473431040 bytes 2024-09-16T12:48:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 480724992 bytes 2024-09-16T12:48:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 525969408 bytes 2024-09-16T12:48:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 535389184 bytes 2024-09-16T12:48:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 598298624 bytes 2024-09-16T12:48:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 642211840 bytes 2024-09-16T12:48:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 689854464 bytes 2024-09-16T12:48:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 773854208 bytes 2024-09-16T12:48:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 823225344 bytes 2024-09-16T12:48:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 871680000 bytes 2024-09-16T12:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 872282112 bytes 2024-09-16T12:48:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 889487360 bytes 2024-09-16T12:48:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 929199104 bytes 2024-09-16T12:48:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 995830784 bytes 2024-09-16T12:48:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1038453760 bytes 2024-09-16T12:48:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1097196544 bytes 2024-09-16T12:48:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1108932608 bytes 2024-09-16T12:48:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1109542912 bytes 2024-09-16T12:48:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1151738880 bytes 2024-09-16T12:48:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1214386176 bytes 2024-09-16T12:48:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1217436672 bytes 2024-09-16T12:48:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1339080704 bytes 2024-09-16T12:48:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1383249920 bytes 2024-09-16T12:48:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1430296576 bytes 2024-09-16T12:48:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1508461568 bytes 2024-09-16T12:48:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1569815552 bytes 2024-09-16T12:48:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1639840768 bytes 2024-09-16T12:48:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1687843840 bytes 2024-09-16T12:48:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1715929088 bytes 2024-09-16T12:48:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1733093376 bytes 2024-09-16T12:48:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1736686592 bytes 2024-09-16T12:48:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1843569664 bytes 2024-09-16T12:48:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1876566016 bytes 2024-09-16T12:48:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1953455104 bytes 2024-09-16T12:48:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1981108224 bytes 2024-09-16T12:48:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2005570560 bytes 2024-09-16T12:48:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2100987904 bytes 2024-09-16T12:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2182122496 bytes 2024-09-16T12:48:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2270513152 bytes 2024-09-16T12:48:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2344543232 bytes 2024-09-16T12:48:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2345782272 bytes 2024-09-16T12:48:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2358492160 bytes 2024-09-16T12:48:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2452951040 bytes 2024-09-16T12:48:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2473216000 bytes 2024-09-16T12:48:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2563508224 bytes 2024-09-16T12:48:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2614342656 bytes 2024-09-16T12:48:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2672168960 bytes 2024-09-16T12:48:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2699023360 bytes 2024-09-16T12:48:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2789085184 bytes 2024-09-16T12:48:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2799389696 bytes 2024-09-16T12:48:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2833114112 bytes 2024-09-16T12:48:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2907586560 bytes 2024-09-16T12:48:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2972856320 bytes 2024-09-16T12:48:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3002562560 bytes 2024-09-16T12:48:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3056350208 bytes 2024-09-16T12:48:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3116329984 bytes 2024-09-16T12:48:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3155304448 bytes 2024-09-16T12:48:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3214971904 bytes 2024-09-16T12:48:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3274419200 bytes 2024-09-16T12:48:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3299120128 bytes 2024-09-16T12:48:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3344310272 bytes 2024-09-16T12:48:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3380016128 bytes 2024-09-16T12:48:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3414920192 bytes 2024-09-16T12:48:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3549659136 bytes 2024-09-16T12:48:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3588719616 bytes 2024-09-16T12:48:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3629573120 bytes 2024-09-16T12:48:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3647120384 bytes 2024-09-16T12:48:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3649361920 bytes 2024-09-16T12:48:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3673719808 bytes 2024-09-16T12:48:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3758120960 bytes 2024-09-16T12:48:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3828356096 bytes 2024-09-16T12:48:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3861940224 bytes 2024-09-16T12:48:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3899326464 bytes 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:48:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:48:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:48:33.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7158710320297639) > weight (0.25) 2024-09-16T12:48:33.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7983129326528906) > weight (0.25) 2024-09-16T12:48:33.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.32246491609506545) > weight (0.25) 2024-09-16T12:48:33.734 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T12:48:33.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T12:48:33.735 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T12:48:33.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T12:48:33.734+0000 7fa5975f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40086) UID: 1000 2024-09-16T12:48:33.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:48:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:48:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:48:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:48:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17855897.6,"bytes_per_snapshot":518130688.0,"last_snapshot_bytes":535676928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490890,"remote_snapshot_timestamp":1726490890,"replay_state":"idle"} 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:48:26 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:48:26' 2024-09-16T12:48:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:48:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:48:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17855897.6,"bytes_per_snapshot":518130688.0,"last_snapshot_bytes":535676928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490890,"remote_snapshot_timestamp":1726490890,"replay_state":"idle"} 2024-09-16T12:48:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:48:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:48:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:48:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:48:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:48:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:48:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:48:26' 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:48:26' 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:48:26' 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":17855897.6,"bytes_per_snapshot":518130688.0,"last_snapshot_bytes":535676928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490890,"remote_snapshot_timestamp":1726490890,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:48:26cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:48:26' 2024-09-16T12:48:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:48:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:48:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:48:26' 2024-09-16T12:48:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:48:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:48:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:48:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:48:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-16T12:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap5 2024-09-16T12:48:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-16T12:48:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:48:42.736 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T12:48:42.736 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:48:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:48:42.736 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T12:48:42.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T12:48:42.737 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T12:48:42.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T12:48:42.740 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-16T12:48:53.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7033510951207297) > weight (0.25) 2024-09-16T12:48:53.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.703091393851743) > weight (0.25) 2024-09-16T12:48:53.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9132839304515878) > weight (0.25) 2024-09-16T12:48:53.728 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5000227458415767) > weight (0.25) 2024-09-16T12:48:53.728 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-16T12:49:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:49:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:49:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:49:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:49:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:49:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:49:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:49:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:49:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:49:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:49:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:49:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:49:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:49:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:49:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:49:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:49:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:49:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:49:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:49:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:49:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:49:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:49:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:49:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:49:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:49:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:49:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:49:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:49:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:49:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:49:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:49:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:49:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:49:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:49:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:49:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:49:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:49:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:49:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:49:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:49:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:49:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:49:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:49:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:49:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:49:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:49:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 21 2024-09-16T12:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:49:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:49:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:49:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:49:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=21 2024-09-16T12:49:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:49:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:49:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:49:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:49:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:49:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:49:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 21 = '' 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:49:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:49:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:49:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:49:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:49:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:49:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:49:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:49:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:49:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 21 = '' 2024-09-16T12:49:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:49:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:49:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:49:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:49:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:49:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:49:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:49:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:49:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:49:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:49:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 21 = '' 2024-09-16T12:49:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:49:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:49:08.666 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:49:08.663+0000 7f6068a65640 -1 reset not still connected to 0x55b6d7fc0c30 2024-09-16T12:49:08.666 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:49:08.663+0000 7f6068a65640 -1 reset not still connected to 0x55b6d81a0f70 2024-09-16T12:49:08.666 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:49:08.663+0000 7f6068a65640 -1 reset not still connected to 0x55b6d81a1790 2024-09-16T12:49:08.666 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:49:08.663+0000 7f6068a65640 -1 reset not still connected to 0x55b6d88291e0 2024-09-16T12:49:08.666 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T12:49:08.663+0000 7f6068a65640 -1 reset not still connected to 0x55b6d8829380 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:49:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:49:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:49:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=21 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 21 = 21 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-16T12:49:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:49:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:49:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:49:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-16T12:49:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:49:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:49:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:49:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 10 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:48:16 2024 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap4-clean ']' 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:49:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:49:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:49:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:49:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:49:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:49:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:49:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:49:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap5-clean 2024-09-16T12:49:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-16T12:49:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-16T12:49:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:49:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:49:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:49:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:49:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T12:49:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:49:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:49:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 51836928 bytes 2024-09-16T12:49:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 127489024 bytes 2024-09-16T12:49:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 128755712 bytes 2024-09-16T12:49:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 190931968 bytes 2024-09-16T12:49:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 226222080 bytes 2024-09-16T12:49:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 288015360 bytes 2024-09-16T12:49:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 306276352 bytes 2024-09-16T12:49:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 320828416 bytes 2024-09-16T12:49:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 399181824 bytes 2024-09-16T12:49:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 444468224 bytes 2024-09-16T12:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 455903232 bytes 2024-09-16T12:49:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 498086912 bytes 2024-09-16T12:49:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 578291712 bytes 2024-09-16T12:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 603447296 bytes 2024-09-16T12:49:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 659367936 bytes 2024-09-16T12:49:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 710835200 bytes 2024-09-16T12:49:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 738708480 bytes 2024-09-16T12:49:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 746436608 bytes 2024-09-16T12:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 764710912 bytes 2024-09-16T12:49:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 795268096 bytes 2024-09-16T12:49:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 831646720 bytes 2024-09-16T12:49:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 858184704 bytes 2024-09-16T12:49:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 938878976 bytes 2024-09-16T12:49:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 951489536 bytes 2024-09-16T12:49:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 971385856 bytes 2024-09-16T12:49:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1001755648 bytes 2024-09-16T12:49:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1032132608 bytes 2024-09-16T12:49:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1098482688 bytes 2024-09-16T12:49:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1099099136 bytes 2024-09-16T12:49:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1158717440 bytes 2024-09-16T12:49:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1212943360 bytes 2024-09-16T12:49:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1257553920 bytes 2024-09-16T12:49:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1302379520 bytes 2024-09-16T12:49:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1326184448 bytes 2024-09-16T12:49:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1348436992 bytes 2024-09-16T12:49:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1365716992 bytes 2024-09-16T12:49:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1372382208 bytes 2024-09-16T12:49:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1427982336 bytes 2024-09-16T12:49:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1495441408 bytes 2024-09-16T12:49:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1520589824 bytes 2024-09-16T12:49:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1538148352 bytes 2024-09-16T12:49:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1588807680 bytes 2024-09-16T12:49:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1665985536 bytes 2024-09-16T12:49:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1709470720 bytes 2024-09-16T12:49:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1724227584 bytes 2024-09-16T12:49:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1755721728 bytes 2024-09-16T12:49:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1814582272 bytes 2024-09-16T12:49:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1911223296 bytes 2024-09-16T12:49:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1956661248 bytes 2024-09-16T12:49:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1980120064 bytes 2024-09-16T12:49:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2065131520 bytes 2024-09-16T12:49:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2123570176 bytes 2024-09-16T12:49:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2170326016 bytes 2024-09-16T12:49:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2187503616 bytes 2024-09-16T12:49:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2224433152 bytes 2024-09-16T12:49:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2276059136 bytes 2024-09-16T12:49:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2366603264 bytes 2024-09-16T12:49:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2407496704 bytes 2024-09-16T12:49:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2491015168 bytes 2024-09-16T12:49:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2526722048 bytes 2024-09-16T12:49:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2538763264 bytes 2024-09-16T12:49:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2569574400 bytes 2024-09-16T12:49:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2626532352 bytes 2024-09-16T12:49:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2654485504 bytes 2024-09-16T12:49:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2756255744 bytes 2024-09-16T12:49:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2793152512 bytes 2024-09-16T12:49:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2821182464 bytes 2024-09-16T12:49:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2891879424 bytes 2024-09-16T12:49:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3009833984 bytes 2024-09-16T12:49:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3064270848 bytes 2024-09-16T12:49:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3079188480 bytes 2024-09-16T12:49:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3097174016 bytes 2024-09-16T12:49:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3116974080 bytes 2024-09-16T12:49:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3143723008 bytes 2024-09-16T12:49:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3186069504 bytes 2024-09-16T12:49:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3213807616 bytes 2024-09-16T12:49:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3266351104 bytes 2024-09-16T12:49:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3308134400 bytes 2024-09-16T12:49:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3338031104 bytes 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:49:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:49:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:49:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:49:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:49:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:49:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:49:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17891942.4,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"} 2024-09-16T12:49:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:49:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:49:26 2024-09-16T12:49:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:49:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:49:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:49:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:49:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:49:26' 2024-09-16T12:49:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:49:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:49:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:49:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17891942.4,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"} 2024-09-16T12:49:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:49:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:49:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:49:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:49:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:49:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:49:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:49:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:49:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:49:26' 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:49:26' 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:49:26' 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:49:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:49:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":17891942.4,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:49:26cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:49:26' 2024-09-16T12:49:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:49:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:49:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:49:26' 2024-09-16T12:49:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:49:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:49:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:49:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:49:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap6 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-16T12:49:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:50:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:50:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:50:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:50:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:50:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:50:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:50:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:50:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:50:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:50:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:50:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:50:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:50:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:50:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:50:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:50:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:50:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:50:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:50:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:50:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:50:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:50:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:50:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:50:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:50:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:50:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:50:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 25 2024-09-16T12:50:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:50:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=25 2024-09-16T12:50:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 25 = '' 2024-09-16T12:50:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:50:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 25 = '' 2024-09-16T12:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:50:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 25 = '' 2024-09-16T12:50:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:50:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=25 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 25 = 25 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-16T12:50:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:50:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:50:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:50:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 12 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:50:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:48:16 2024 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:50:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:50:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap5-clean ']' 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-16T12:50:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:50:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap6-clean 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-16T12:50:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:50:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T12:50:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:50:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:50:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 67169280 bytes 2024-09-16T12:50:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 149552128 bytes 2024-09-16T12:50:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 194722816 bytes 2024-09-16T12:50:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 233936896 bytes 2024-09-16T12:50:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 317674496 bytes 2024-09-16T12:50:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 367252480 bytes 2024-09-16T12:50:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 410840064 bytes 2024-09-16T12:50:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 413655040 bytes 2024-09-16T12:50:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 464353280 bytes 2024-09-16T12:50:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 518047744 bytes 2024-09-16T12:50:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 525280256 bytes 2024-09-16T12:50:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 552978432 bytes 2024-09-16T12:50:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 591539200 bytes 2024-09-16T12:50:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 599954432 bytes 2024-09-16T12:50:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 610999296 bytes 2024-09-16T12:50:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 666375168 bytes 2024-09-16T12:50:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 716944384 bytes 2024-09-16T12:50:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 761288704 bytes 2024-09-16T12:50:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 804531200 bytes 2024-09-16T12:50:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 847745024 bytes 2024-09-16T12:50:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 856324096 bytes 2024-09-16T12:50:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 862353408 bytes 2024-09-16T12:50:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 879223808 bytes 2024-09-16T12:50:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 910820352 bytes 2024-09-16T12:50:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 954950656 bytes 2024-09-16T12:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 978022400 bytes 2024-09-16T12:50:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 996526080 bytes 2024-09-16T12:50:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1030844416 bytes 2024-09-16T12:50:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1044089856 bytes 2024-09-16T12:50:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1077510144 bytes 2024-09-16T12:50:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1097071616 bytes 2024-09-16T12:50:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1144365056 bytes 2024-09-16T12:50:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1188689920 bytes 2024-09-16T12:50:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1203014656 bytes 2024-09-16T12:50:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1264545792 bytes 2024-09-16T12:50:15.741 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T12:50:15.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T12:50:15.742 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T12:50:15.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T12:50:15.740+0000 7fca1d1f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38158) UID: 1000 2024-09-16T12:50:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1277713408 bytes 2024-09-16T12:50:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1286048768 bytes 2024-09-16T12:50:15.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1293899776 bytes 2024-09-16T12:50:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1345300480 bytes 2024-09-16T12:50:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1382556672 bytes 2024-09-16T12:50:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1452733440 bytes 2024-09-16T12:50:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1484002304 bytes 2024-09-16T12:50:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1525875712 bytes 2024-09-16T12:50:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1578796032 bytes 2024-09-16T12:50:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1637189632 bytes 2024-09-16T12:50:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1680211968 bytes 2024-09-16T12:50:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1690995712 bytes 2024-09-16T12:50:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1716273152 bytes 2024-09-16T12:50:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1759536128 bytes 2024-09-16T12:50:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1806040064 bytes 2024-09-16T12:50:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1806647296 bytes 2024-09-16T12:50:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1846927360 bytes 2024-09-16T12:50:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1866285056 bytes 2024-09-16T12:50:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1950434304 bytes 2024-09-16T12:50:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2032653312 bytes 2024-09-16T12:50:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2033255424 bytes 2024-09-16T12:50:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2115632128 bytes 2024-09-16T12:50:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2129426432 bytes 2024-09-16T12:50:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2191012864 bytes 2024-09-16T12:50:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2227637248 bytes 2024-09-16T12:50:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2270567424 bytes 2024-09-16T12:50:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2298796032 bytes 2024-09-16T12:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2346812416 bytes 2024-09-16T12:50:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2385406976 bytes 2024-09-16T12:50:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2405143552 bytes 2024-09-16T12:50:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2498521088 bytes 2024-09-16T12:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:50:20.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.333534981396976) > weight (0.25) 2024-09-16T12:50:20.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7755641670878517) > weight (0.25) 2024-09-16T12:50:20.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4425752419759612) > weight (0.25) 2024-09-16T12:50:20.729 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.44353927844721963) > weight (0.25) 2024-09-16T12:50:20.729 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-16T12:50:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:50:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:50:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:50:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:50:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:50:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:50:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"} 2024-09-16T12:50:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:49:56 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+stopped 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:49:56' 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:50:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:50:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:50:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"} 2024-09-16T12:50:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:50:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:50:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:50:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:50:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:50:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:50:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:50:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:49:56' 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:49:56' 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:49:56' 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:50:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:50:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='WARNINGOKWARNING142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536217600.0,"last_snapshot_bytes":536758272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726490945,"remote_snapshot_timestamp":1726490945,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:49:56cluster268392c8a-786e-4db9-983c-eb2f365ca1cadown+stoppedlocal image is primary2024-09-16 12:49:56' 2024-09-16T12:50:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:50:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:50:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:49:56' 2024-09-16T12:50:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:50:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:50:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:50:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:50:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:50:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-16T12:50:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap7 2024-09-16T12:50:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-16T12:50:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:50:22.743 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T12:50:22.743 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:50:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:50:22.744 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T12:50:22.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T12:50:22.744 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T12:50:22.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T12:50:22.748 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-16T12:50:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:50:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:50:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:50:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:50:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:50:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:50:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:50:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:50:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:50:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:50:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:50:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:50:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:50:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:50:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:50:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:50:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:50:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:50:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:50:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:50:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:50:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:50:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:50:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:50:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:50:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:50:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:50:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:50:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:50:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:50:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:50:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:50:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:50:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:50:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:50:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:50:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:50:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:50:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:50:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:50:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:50:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:50:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:50:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:50:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:50:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:50:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:50:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:50:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:50:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:50:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:50:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:50:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:50:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:50:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 29 2024-09-16T12:50:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:50:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:50:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:50:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=29 2024-09-16T12:50:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:50:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:50:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-16T12:50:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:50:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:50:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:50:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:50:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:50:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-16T12:50:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:50:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:50:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:50:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:50:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:50:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=29 2024-09-16T12:50:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = 29 2024-09-16T12:50:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-16T12:50:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:50:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:50:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-16T12:50:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:50:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 14 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:50:07 2024 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:50:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:50:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:50:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:50:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap6-clean ']' 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:50:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:50:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:50:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap7-clean 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-16T12:50:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:50:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:50:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:50:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:50:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:50:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:50:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:50:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-16T12:50:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:50:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:50:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 21339136 bytes 2024-09-16T12:50:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 78092288 bytes 2024-09-16T12:51:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 126264320 bytes 2024-09-16T12:51:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 128074752 bytes 2024-09-16T12:51:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 144668672 bytes 2024-09-16T12:51:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 231772160 bytes 2024-09-16T12:51:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 325968896 bytes 2024-09-16T12:51:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 352784384 bytes 2024-09-16T12:51:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 388270080 bytes 2024-09-16T12:51:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 441791488 bytes 2024-09-16T12:51:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 505604096 bytes 2024-09-16T12:51:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 584493056 bytes 2024-09-16T12:51:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 620797952 bytes 2024-09-16T12:51:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 672212992 bytes 2024-09-16T12:51:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 672826368 bytes 2024-09-16T12:51:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 703742976 bytes 2024-09-16T12:51:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 732934144 bytes 2024-09-16T12:51:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 784561152 bytes 2024-09-16T12:51:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 832005120 bytes 2024-09-16T12:51:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 859014144 bytes 2024-09-16T12:51:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 918681600 bytes 2024-09-16T12:51:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 969726976 bytes 2024-09-16T12:51:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1057158144 bytes 2024-09-16T12:51:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1071713280 bytes 2024-09-16T12:51:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1112622080 bytes 2024-09-16T12:51:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1165252608 bytes 2024-09-16T12:51:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1232632832 bytes 2024-09-16T12:51:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1279996928 bytes 2024-09-16T12:51:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1313044480 bytes 2024-09-16T12:51:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1359091712 bytes 2024-09-16T12:51:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1388569600 bytes 2024-09-16T12:51:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1422393344 bytes 2024-09-16T12:51:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1455937536 bytes 2024-09-16T12:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1476023296 bytes 2024-09-16T12:51:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1534514176 bytes 2024-09-16T12:51:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1608500224 bytes 2024-09-16T12:51:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1637365760 bytes 2024-09-16T12:51:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1706562560 bytes 2024-09-16T12:51:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1739781120 bytes 2024-09-16T12:51:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1768103936 bytes 2024-09-16T12:51:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1786288128 bytes 2024-09-16T12:51:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1831702528 bytes 2024-09-16T12:51:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1875023872 bytes 2024-09-16T12:51:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1921846272 bytes 2024-09-16T12:51:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1959802880 bytes 2024-09-16T12:51:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1960410112 bytes 2024-09-16T12:51:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2003098624 bytes 2024-09-16T12:51:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2030418944 bytes 2024-09-16T12:51:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2094638080 bytes 2024-09-16T12:51:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:51:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:51:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:51:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17744179.2,"bytes_per_snapshot":534535680.0,"last_snapshot_bytes":532325376,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491050,"remote_snapshot_timestamp":1726491050,"replay_state":"idle"} 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:50:56 2024-09-16T12:51:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:50:56' 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:51:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:51:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:51:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17744179.2,"bytes_per_snapshot":534535680.0,"last_snapshot_bytes":532325376,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491050,"remote_snapshot_timestamp":1726491050,"replay_state":"idle"} 2024-09-16T12:51:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:51:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:51:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:51:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:51:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:51:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:51:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:51:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:50:56' 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:50:56' 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:51:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:50:56' 2024-09-16T12:51:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:51:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:51:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:51:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42264298mirror.2smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":17744179.2,"bytes_per_snapshot":534535680.0,"last_snapshot_bytes":532325376,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491050,"remote_snapshot_timestamp":1726491050,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:50:56cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:50:56' 2024-09-16T12:51:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:51:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:51:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:50:56' 2024-09-16T12:51:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:51:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:51:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:51:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:51:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap8 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-16T12:51:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:51:29.730 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5086980392455775) > weight (0.25) 2024-09-16T12:51:29.731 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7495023671056115) > weight (0.25) 2024-09-16T12:51:29.731 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T12:51:29.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T12:51:29.731 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T12:51:29.732 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T12:51:29.732 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:51:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:51:29.731+0000 7f909a3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38104) UID: 1000 2024-09-16T12:51:29.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:51:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:51:35.833 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T12:51:35.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T12:51:35.834 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T12:51:35.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T12:51:35.837 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-16T12:51:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:51:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:51:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:51:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:51:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:51:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:51:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:51:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:51:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:51:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:51:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:51:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:51:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:51:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:51:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:51:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:51:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:51:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:51:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:51:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:51:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:51:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:51:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:51:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:51:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:51:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:51:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:51:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:51:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:51:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:51:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:51:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:51:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:51:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:51:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:51:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:51:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:51:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:51:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:51:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:51:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:51:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:51:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:51:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:51:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:51:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:51:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:51:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:51:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:51:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:51:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:51:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:51:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:51:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:51:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:51:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:51:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:51:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:51:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 33 2024-09-16T12:51:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:51:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:51:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:51:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:51:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:51:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=33 2024-09-16T12:51:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:51:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:51:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:51:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:51:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:51:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:51:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:51:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:51:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:51:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:51:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 33 = '' 2024-09-16T12:51:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:51:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:51:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:51:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:51:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:51:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 33 = '' 2024-09-16T12:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:51:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:51:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:51:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 33 = '' 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:51:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:51:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:51:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:51:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:51:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=33 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 33 = 33 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:51:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-16T12:51:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:51:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:51:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:51:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 16 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:51:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:50:07 2024 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:51:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap7-clean ']' 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:51:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:51:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:51:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:51:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:51:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap8-clean 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-16T12:51:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:51:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:51:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T12:51:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:51:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:51:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 29650944 bytes 2024-09-16T12:51:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 57938944 bytes 2024-09-16T12:51:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 120073216 bytes 2024-09-16T12:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 211025920 bytes 2024-09-16T12:51:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 230598656 bytes 2024-09-16T12:51:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 378919936 bytes 2024-09-16T12:51:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 419497984 bytes 2024-09-16T12:51:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 439824384 bytes 2024-09-16T12:51:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 440418304 bytes 2024-09-16T12:51:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 452768768 bytes 2024-09-16T12:51:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 504160256 bytes 2024-09-16T12:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 528865280 bytes 2024-09-16T12:51:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 577321984 bytes 2024-09-16T12:51:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 635993088 bytes 2024-09-16T12:51:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 680688640 bytes 2024-09-16T12:51:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 708979712 bytes 2024-09-16T12:51:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 713421824 bytes 2024-09-16T12:51:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 767106048 bytes 2024-09-16T12:51:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 776118272 bytes 2024-09-16T12:51:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 799537152 bytes 2024-09-16T12:51:52.748 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T12:51:52.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T12:51:52.749 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T12:51:52.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T12:51:52.747+0000 7f5831bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 43190) UID: 1000 2024-09-16T12:51:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 831505408 bytes 2024-09-16T12:51:52.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:51:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 857291776 bytes 2024-09-16T12:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 861990912 bytes 2024-09-16T12:51:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 904750080 bytes 2024-09-16T12:51:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1011062784 bytes 2024-09-16T12:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1055804416 bytes 2024-09-16T12:51:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1101186048 bytes 2024-09-16T12:51:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1132224512 bytes 2024-09-16T12:51:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1175098368 bytes 2024-09-16T12:51:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1213487104 bytes 2024-09-16T12:51:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1255019520 bytes 2024-09-16T12:51:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1291801600 bytes 2024-09-16T12:51:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1343538176 bytes 2024-09-16T12:51:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1345486848 bytes 2024-09-16T12:51:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1349740544 bytes 2024-09-16T12:51:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1369879552 bytes 2024-09-16T12:51:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1399267328 bytes 2024-09-16T12:51:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1441164288 bytes 2024-09-16T12:51:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1478779904 bytes 2024-09-16T12:51:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1535839232 bytes 2024-09-16T12:51:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1578699776 bytes 2024-09-16T12:51:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1603598336 bytes 2024-09-16T12:51:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1642909696 bytes 2024-09-16T12:51:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1698856960 bytes 2024-09-16T12:51:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1767218176 bytes 2024-09-16T12:51:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1793507328 bytes 2024-09-16T12:51:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1839352832 bytes 2024-09-16T12:51:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1912071168 bytes 2024-09-16T12:51:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1972187136 bytes 2024-09-16T12:51:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1972809728 bytes 2024-09-16T12:51:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2044079104 bytes 2024-09-16T12:51:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2059302912 bytes 2024-09-16T12:51:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2097810432 bytes 2024-09-16T12:51:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2157989888 bytes 2024-09-16T12:51:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2179958784 bytes 2024-09-16T12:51:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2274131968 bytes 2024-09-16T12:51:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2286438400 bytes 2024-09-16T12:51:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2318111744 bytes 2024-09-16T12:51:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2346954752 bytes 2024-09-16T12:51:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2399782912 bytes 2024-09-16T12:51:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2423589888 bytes 2024-09-16T12:51:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2495597568 bytes 2024-09-16T12:51:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2514607104 bytes 2024-09-16T12:51:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2515211264 bytes 2024-09-16T12:51:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2564559872 bytes 2024-09-16T12:51:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2623341568 bytes 2024-09-16T12:52:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2690900992 bytes 2024-09-16T12:52:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2701477888 bytes 2024-09-16T12:52:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2756629504 bytes 2024-09-16T12:52:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2777149440 bytes 2024-09-16T12:52:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2811796480 bytes 2024-09-16T12:52:00.750 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T12:52:00.750 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:52:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:52:00.751 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T12:52:00.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T12:52:00.751 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T12:52:00.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T12:52:00.755 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-16T12:52:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2864691200 bytes 2024-09-16T12:52:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2908366848 bytes 2024-09-16T12:52:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2908997632 bytes 2024-09-16T12:52:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2974821376 bytes 2024-09-16T12:52:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3000454144 bytes 2024-09-16T12:52:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3025033216 bytes 2024-09-16T12:52:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3036447744 bytes 2024-09-16T12:52:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3135583232 bytes 2024-09-16T12:52:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3148187648 bytes 2024-09-16T12:52:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3237731328 bytes 2024-09-16T12:52:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3277045760 bytes 2024-09-16T12:52:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3349445632 bytes 2024-09-16T12:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3406673920 bytes 2024-09-16T12:52:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3465791488 bytes 2024-09-16T12:52:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3499572224 bytes 2024-09-16T12:52:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:52:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:52:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:52:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:52:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:52:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:52:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:52:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:52:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:52:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:52:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:52:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17106158.93,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"} 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:51:56 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:52:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:51:56' 2024-09-16T12:52:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:52:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:52:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17106158.93,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"} 2024-09-16T12:52:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:52:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:52:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:52:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:52:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:52:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:52:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:52:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:52:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:51:56' 2024-09-16T12:52:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:51:56' 2024-09-16T12:52:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:51:56' 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:52:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:52:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:52:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOK45734576mirror.2smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":17106158.93,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:51:56cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:51:56' 2024-09-16T12:52:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:52:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:52:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:51:56' 2024-09-16T12:52:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:52:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:52:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:52:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:52:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:52:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap9 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-16T12:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:52:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:52:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:52:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:52:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:52:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:52:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:52:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:52:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:52:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:52:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:52:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:52:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:52:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:52:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:52:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:52:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:52:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:52:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:52:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:52:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:52:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:52:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:52:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:52:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:52:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:52:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:52:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:52:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:52:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:52:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:52:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:52:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:52:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:52:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:52:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:52:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:52:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:52:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:52:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:52:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:52:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:52:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:52:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:52:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:52:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:52:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:52:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:52:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:52:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:52:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:52:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:52:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:52:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:52:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:52:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:52:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:52:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:52:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:52:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:52:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:52:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:52:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:52:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:52:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:52:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 37 2024-09-16T12:52:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:52:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:52:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:52:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:52:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:52:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=37 2024-09-16T12:52:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:52:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:52:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:52:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:52:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:52:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:52:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:52:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:52:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:52:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:52:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 37 = '' 2024-09-16T12:52:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:52:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:52:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:52:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:52:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:52:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:52:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:52:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:52:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:52:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 37 = '' 2024-09-16T12:52:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:52:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:52:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:52:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:52:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:52:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:52:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 37 = '' 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:52:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:52:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:52:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=37 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 37 = 37 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:52:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-16T12:52:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:52:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:52:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:52:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 18 2024-09-16T12:52:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:51:45 2024 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:52:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:52:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap8-clean ']' 2024-09-16T12:52:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T12:52:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:52:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:52:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:52:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:52:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:52:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:52:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap9-clean 2024-09-16T12:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-16T12:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-16T12:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:52:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:52:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T12:52:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:52:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:52:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:52:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T12:52:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39603200 bytes 2024-09-16T12:52:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 85809152 bytes 2024-09-16T12:52:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 86418432 bytes 2024-09-16T12:52:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 104206336 bytes 2024-09-16T12:52:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 139551744 bytes 2024-09-16T12:52:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 140151808 bytes 2024-09-16T12:52:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 183272448 bytes 2024-09-16T12:52:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 255827968 bytes 2024-09-16T12:52:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 301187072 bytes 2024-09-16T12:52:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 323993600 bytes 2024-09-16T12:52:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 349743104 bytes 2024-09-16T12:52:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 418331648 bytes 2024-09-16T12:52:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 471931904 bytes 2024-09-16T12:52:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 543165440 bytes 2024-09-16T12:52:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 560885760 bytes 2024-09-16T12:52:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 561499136 bytes 2024-09-16T12:52:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 606620672 bytes 2024-09-16T12:52:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 636022784 bytes 2024-09-16T12:52:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 675719168 bytes 2024-09-16T12:52:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 676357120 bytes 2024-09-16T12:52:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 706746368 bytes 2024-09-16T12:52:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 730180608 bytes 2024-09-16T12:52:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 798633984 bytes 2024-09-16T12:52:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 816030720 bytes 2024-09-16T12:52:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 858016768 bytes 2024-09-16T12:52:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 874011648 bytes 2024-09-16T12:52:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 911227904 bytes 2024-09-16T12:52:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 961784832 bytes 2024-09-16T12:52:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 989473792 bytes 2024-09-16T12:52:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1068178432 bytes 2024-09-16T12:52:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1122777088 bytes 2024-09-16T12:52:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1201167360 bytes 2024-09-16T12:52:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1245572096 bytes 2024-09-16T12:52:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1267826688 bytes 2024-09-16T12:52:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1324327936 bytes 2024-09-16T12:52:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1337038848 bytes 2024-09-16T12:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1429437440 bytes 2024-09-16T12:52:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1467199488 bytes 2024-09-16T12:52:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1518609408 bytes 2024-09-16T12:52:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1538722816 bytes 2024-09-16T12:52:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1590524928 bytes 2024-09-16T12:52:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1668163584 bytes 2024-09-16T12:52:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1728982016 bytes 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:52:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:52:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:52:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-16T12:52:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-16T12:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:52:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T12:52:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"} 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.3 on smithi191 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:52:26 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-16 12:52:26' 2024-09-16T12:52:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:52:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-16T12:52:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-16T12:52:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"} 2024-09-16T12:52:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:52:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:52:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-16T12:52:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:52:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:52:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.3 on smithi191 2024-09-16T12:52:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-16T12:52:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:52:26' 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 12:52:26' 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-16 12:52:26' 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-16T12:52:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:52:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142224302mirror.1smithi19117.2.7-1643-gd75f2382falseOK42304297mirror.3smithi19117.2.7-1643-gd75f2382trueOK44134416mirror.0smithi19117.2.7-1643-gd75f2382falseOK45734576mirror.2smithi19117.2.7-1643-gd75f2382falseOKtest868f4cf4-e2b2-4b95-9f45-604f88ffc305up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522755072.0,"last_snapshot_bytes":513184768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726491099,"remote_snapshot_timestamp":1726491099,"replay_state":"idle"}42304297mirror.3smithi1912024-09-16 12:52:26cluster268392c8a-786e-4db9-983c-eb2f365ca1caup+stoppedlocal image is primary2024-09-16 12:52:26' 2024-09-16T12:52:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-16T12:52:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T12:52:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-16 12:52:26' 2024-09-16T12:52:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T12:52:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-16T12:52:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-16T12:52:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-16T12:52:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-16T12:52:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:52:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap10 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-16T12:52:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:53:14.838 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6023070028996916) > weight (0.25) 2024-09-16T12:53:14.838 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8357701483927091) > weight (0.25) 2024-09-16T12:53:14.838 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T12:53:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T12:53:14.839 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T12:53:14.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:53:14.838+0000 7f2a4e3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 43703) UID: 1000 2024-09-16T12:53:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:53:19.840 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T12:53:19.840 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:53:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:53:19.841 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T12:53:19.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T12:53:19.841 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T12:53:19.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T12:53:19.844 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:53:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:53:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:53:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:53:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:53:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:53:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:53:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:53:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:53:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:53:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:53:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:53:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:53:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:53:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:53:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:53:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:53:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:53:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:53:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:53:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:53:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:53:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:53:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:53:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:53:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:53:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:53:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:53:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:53:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:53:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:53:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:53:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:53:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:53:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:53:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:53:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 41 2024-09-16T12:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=41 2024-09-16T12:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:53:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:53:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:53:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:53:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 41 = '' 2024-09-16T12:53:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:53:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 41 = '' 2024-09-16T12:53:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T12:53:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:53:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:53:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 41 = '' 2024-09-16T12:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T12:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=41 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 41 = 41 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-16T12:53:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:53:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:53:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:53:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 20 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:51:45 2024 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:53:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:53:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap9-clean ']' 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap10-clean 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-16T12:53:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T12:53:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:53:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:53:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T12:53:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T12:53:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T12:53:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:53:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T12:53:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T12:53:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T12:53:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T12:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T12:53:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T12:53:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T12:53:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T12:53:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T12:53:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T12:53:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T12:53:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T12:53:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T12:53:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T12:53:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T12:53:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T12:53:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T12:53:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T12:53:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T12:53:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T12:53:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T12:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T12:53:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T12:53:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T12:53:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T12:53:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T12:53:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T12:53:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T12:53:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T12:53:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T12:53:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T12:53:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T12:53:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T12:53:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T12:53:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T12:53:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T12:53:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T12:53:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-16T12:53:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T12:53:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 44 2024-09-16T12:53:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:53:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.status 2024-09-16T12:53:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T12:53:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:35.755 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T12:53:35.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T12:53:35.755 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T12:53:35.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T12:53:35.755+0000 7f84f39f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 44153) UID: 1000 2024-09-16T12:53:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T12:53:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=44 2024-09-16T12:53:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T12:53:35.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:53:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T12:53:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T12:53:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T12:53:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T12:53:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T12:53:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 44 = '' 2024-09-16T12:53:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T12:53:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.status 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T12:53:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T12:53:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T12:53:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=44 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 44 = 44 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T12:53:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T12:53:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 21 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c96d9218d7 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c96d9218d7 2024-09-16T12:53:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:53:29 2024 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:44:58 2024 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 868f4cf4-e2b2-4b95-9f45-604f88ffc305 2024-09-16T12:53:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T12:53:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1-clean 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2-clean 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:44.756 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T12:53:44.756 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:53:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:53:44.757 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T12:53:44.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T12:53:44.757 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T12:53:44.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T12:53:44.760 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-16T12:53:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3-clean 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-16T12:53:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4-clean 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-16T12:53:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T12:53:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5-clean 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6-clean 2024-09-16T12:53:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T12:53:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:53:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:53:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:53:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7-clean 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-16T12:54:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8-clean 2024-09-16T12:54:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T12:54:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9-clean 2024-09-16T12:54:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10-clean 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export 2024-09-16T12:54:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T12:54:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-test.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-test.export 2024-09-16T12:54:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap1 2024-09-16T12:54:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-16T12:54:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1-clean 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap1-clean 2024-09-16T12:54:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-16T12:54:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap2 2024-09-16T12:54:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-16T12:54:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2-clean 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap2-clean 2024-09-16T12:54:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-16T12:54:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap3 2024-09-16T12:54:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-16T12:54:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3-clean 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap3-clean 2024-09-16T12:54:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-16T12:54:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap4 2024-09-16T12:54:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-16T12:54:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4-clean 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap4-clean 2024-09-16T12:54:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-16T12:54:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap5 2024-09-16T12:54:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-16T12:54:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5-clean 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap5-clean 2024-09-16T12:54:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-16T12:54:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap6 2024-09-16T12:54:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-16T12:54:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6-clean 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap6-clean 2024-09-16T12:54:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-16T12:54:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap7 2024-09-16T12:54:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-16T12:54:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7-clean 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap7-clean 2024-09-16T12:54:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-16T12:54:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap8 2024-09-16T12:54:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-16T12:54:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8-clean 2024-09-16T12:54:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-16T12:54:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap8-clean 2024-09-16T12:54:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-16T12:54:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap9 2024-09-16T12:54:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-16T12:54:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9-clean 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap9-clean 2024-09-16T12:54:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-16T12:54:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap10 2024-09-16T12:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-16T12:54:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10-clean 2024-09-16T12:54:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-16T12:54:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap10-clean 2024-09-16T12:54:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-16T12:54:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror test 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T12:54:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T12:54:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror test 2024-09-16T12:54:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-16T12:54:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-16T12:54:31.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T12:54:31.806+0000 7f45eebeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5608a8e4aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T12:54:31.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T12:54:31.806+0000 7f45f53f8640 -1 rbd::mirror::ImageReplayer: 0x5608a68dac80 [3/868f4cf4-e2b2-4b95-9f45-604f88ffc305] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T12:54:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-16T12:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror test 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=test 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T12:54:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T12:54:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T12:54:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T12:54:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T12:54:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T12:54:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-16T12:54:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T12:54:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror test 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=test 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T12:54:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T12:54:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: create many images' 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T12:54:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T12:54:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 12:54:33 TEST: create many images 2024-09-16T12:54:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 12:54:33 TEST: create many images 2024-09-16T12:54:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap 2024-09-16T12:54:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-16T12:54:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-16T12:54:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-09-16T12:54:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T12:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:54:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T12:54:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1281e07c8930 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1281e07c8930 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:33 2024 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:33 2024 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:33 2024 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:54:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:54:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 70a8076e-7461-4223-9205-c586ea82527e 2024-09-16T12:54:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:54:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:54:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T12:54:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T12:54:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:54:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T12:54:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:54:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:54:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 17715200 bytes 2024-09-16T12:54:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 87122944 bytes 2024-09-16T12:54:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 127365120 bytes 2024-09-16T12:54:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 176289792 bytes 2024-09-16T12:54:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 274283520 bytes 2024-09-16T12:54:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 333673472 bytes 2024-09-16T12:54:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 379957248 bytes 2024-09-16T12:54:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 458957824 bytes 2024-09-16T12:54:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 460805120 bytes 2024-09-16T12:54:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 565608448 bytes 2024-09-16T12:54:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 608736256 bytes 2024-09-16T12:54:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 637444096 bytes 2024-09-16T12:54:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 670457856 bytes 2024-09-16T12:54:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 683006976 bytes 2024-09-16T12:54:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 719206400 bytes 2024-09-16T12:54:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 749922304 bytes 2024-09-16T12:54:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 785110016 bytes 2024-09-16T12:54:36.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 830697472 bytes 2024-09-16T12:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 890999808 bytes 2024-09-16T12:54:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 928911360 bytes 2024-09-16T12:54:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 988853248 bytes 2024-09-16T12:54:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1022276608 bytes 2024-09-16T12:54:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1043377152 bytes 2024-09-16T12:54:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1089291264 bytes 2024-09-16T12:54:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1132062720 bytes 2024-09-16T12:54:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1146415104 bytes 2024-09-16T12:54:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1155742720 bytes 2024-09-16T12:54:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1211185152 bytes 2024-09-16T12:54:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1232592896 bytes 2024-09-16T12:54:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1289286656 bytes 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:54:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:54:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T12:54:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:54:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T12:54:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 128dec010afd 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.128dec010afd 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:38 2024 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:38 2024 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:38 2024 2024-09-16T12:54:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:54:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:54:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5d32e8f9-d6e5-4ee3-9c1a-b169f80cfbb0 2024-09-16T12:54:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T12:54:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:54:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T12:54:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:54:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:54:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 79016960 bytes 2024-09-16T12:54:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 152023040 bytes 2024-09-16T12:54:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 184556544 bytes 2024-09-16T12:54:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 237421568 bytes 2024-09-16T12:54:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 290554880 bytes 2024-09-16T12:54:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 320245760 bytes 2024-09-16T12:54:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 382361600 bytes 2024-09-16T12:54:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 449270784 bytes 2024-09-16T12:54:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 484167680 bytes 2024-09-16T12:54:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 510435328 bytes 2024-09-16T12:54:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 612645888 bytes 2024-09-16T12:54:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 654722048 bytes 2024-09-16T12:54:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 691964928 bytes 2024-09-16T12:54:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 713043968 bytes 2024-09-16T12:54:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 778826752 bytes 2024-09-16T12:54:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 815587328 bytes 2024-09-16T12:54:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 878052352 bytes 2024-09-16T12:54:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 913053696 bytes 2024-09-16T12:54:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 991825920 bytes 2024-09-16T12:54:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1035096064 bytes 2024-09-16T12:54:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1074612224 bytes 2024-09-16T12:54:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1171286016 bytes 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1202082816 bytes 2024-09-16T12:54:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1202705408 bytes 2024-09-16T12:54:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1273676800 bytes 2024-09-16T12:54:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1303093248 bytes 2024-09-16T12:54:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1326493696 bytes 2024-09-16T12:54:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1351285760 bytes 2024-09-16T12:54:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1449578496 bytes 2024-09-16T12:54:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1498492928 bytes 2024-09-16T12:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1542270976 bytes 2024-09-16T12:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1560093696 bytes 2024-09-16T12:54:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:54:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:54:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:54:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-09-16T12:54:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:54:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-16T12:54:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T12:54:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T12:54:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:54:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T12:54:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-16T12:54:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:54:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:54:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:54:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129950a6cb7e 2024-09-16T12:54:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129950a6cb7e 2024-09-16T12:54:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:54:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:45 2024 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:45 2024 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:45 2024 2024-09-16T12:54:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:54:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:54:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: fa91e86c-1fbe-49d8-980f-ba4716f8e375 2024-09-16T12:54:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T12:54:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T12:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:54:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:54:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 36781056 bytes 2024-09-16T12:54:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 58919936 bytes 2024-09-16T12:54:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 99534848 bytes 2024-09-16T12:54:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 165468160 bytes 2024-09-16T12:54:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 208785408 bytes 2024-09-16T12:54:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 231488512 bytes 2024-09-16T12:54:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 262077440 bytes 2024-09-16T12:54:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 301156352 bytes 2024-09-16T12:54:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 395271168 bytes 2024-09-16T12:54:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 431303680 bytes 2024-09-16T12:54:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 433214464 bytes 2024-09-16T12:54:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 433823744 bytes 2024-09-16T12:54:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 451755008 bytes 2024-09-16T12:54:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 528216064 bytes 2024-09-16T12:54:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 534926336 bytes 2024-09-16T12:54:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 559813632 bytes 2024-09-16T12:54:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 571507712 bytes 2024-09-16T12:54:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 587064320 bytes 2024-09-16T12:54:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 601990144 bytes 2024-09-16T12:54:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 641171456 bytes 2024-09-16T12:54:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 684342272 bytes 2024-09-16T12:54:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 719223808 bytes 2024-09-16T12:54:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 739584000 bytes 2024-09-16T12:54:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 752656384 bytes 2024-09-16T12:54:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 828574720 bytes 2024-09-16T12:54:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 908213248 bytes 2024-09-16T12:54:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 974764032 bytes 2024-09-16T12:54:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1016281088 bytes 2024-09-16T12:54:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1017987072 bytes 2024-09-16T12:54:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1044813824 bytes 2024-09-16T12:54:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1086690304 bytes 2024-09-16T12:54:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1102024704 bytes 2024-09-16T12:54:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1171949568 bytes 2024-09-16T12:54:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1222026240 bytes 2024-09-16T12:54:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1288515584 bytes 2024-09-16T12:54:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1301903360 bytes 2024-09-16T12:54:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1366416384 bytes 2024-09-16T12:54:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1380454400 bytes 2024-09-16T12:54:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1418092544 bytes 2024-09-16T12:54:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1430984704 bytes 2024-09-16T12:54:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1455735808 bytes 2024-09-16T12:54:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1484867584 bytes 2024-09-16T12:54:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1485489152 bytes 2024-09-16T12:54:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1539163136 bytes 2024-09-16T12:54:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1582523392 bytes 2024-09-16T12:54:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1601298432 bytes 2024-09-16T12:54:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1630700544 bytes 2024-09-16T12:54:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1672619008 bytes 2024-09-16T12:54:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1723652096 bytes 2024-09-16T12:54:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1761850368 bytes 2024-09-16T12:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1808570368 bytes 2024-09-16T12:54:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1833825280 bytes 2024-09-16T12:54:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1863341056 bytes 2024-09-16T12:54:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1910666240 bytes 2024-09-16T12:54:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 1925922816 bytes 2024-09-16T12:54:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 1955659776 bytes 2024-09-16T12:54:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 1997006848 bytes 2024-09-16T12:54:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2008611840 bytes 2024-09-16T12:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2018903040 bytes 2024-09-16T12:54:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2052786176 bytes 2024-09-16T12:54:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2086299648 bytes 2024-09-16T12:54:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2117603328 bytes 2024-09-16T12:54:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2169572352 bytes 2024-09-16T12:54:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2196269056 bytes 2024-09-16T12:54:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2196902912 bytes 2024-09-16T12:54:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2250862592 bytes 2024-09-16T12:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2299345920 bytes 2024-09-16T12:54:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2339279872 bytes 2024-09-16T12:54:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2351334400 bytes 2024-09-16T12:54:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2354229248 bytes 2024-09-16T12:54:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2378071040 bytes 2024-09-16T12:54:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2387364864 bytes 2024-09-16T12:54:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2411874304 bytes 2024-09-16T12:54:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2452446208 bytes 2024-09-16T12:54:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2464457728 bytes 2024-09-16T12:54:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2520433664 bytes 2024-09-16T12:54:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 2540858368 bytes 2024-09-16T12:54:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 2572676096 bytes 2024-09-16T12:54:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 2662739968 bytes 2024-09-16T12:54:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 2692323328 bytes 2024-09-16T12:54:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 2748444672 bytes 2024-09-16T12:54:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 2797816832 bytes 2024-09-16T12:54:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 2892677120 bytes 2024-09-16T12:54:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 2935359488 bytes 2024-09-16T12:54:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:54:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:54:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:54:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:54:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:54:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T12:55:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12a5cad17b30 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12a5cad17b30 2024-09-16T12:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:59 2024 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:59 2024 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:59 2024 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 9f626ce6-5b6e-44bf-be4c-04da712913b0 2024-09-16T12:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T12:55:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T12:55:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T12:55:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 36263936 bytes 2024-09-16T12:55:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 135161856 bytes 2024-09-16T12:55:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 148439040 bytes 2024-09-16T12:55:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 241633280 bytes 2024-09-16T12:55:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 242266112 bytes 2024-09-16T12:55:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 274575360 bytes 2024-09-16T12:55:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 280289280 bytes 2024-09-16T12:55:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 353764352 bytes 2024-09-16T12:55:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 385861632 bytes 2024-09-16T12:55:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 402128896 bytes 2024-09-16T12:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 474707968 bytes 2024-09-16T12:55:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 499381248 bytes 2024-09-16T12:55:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 559030272 bytes 2024-09-16T12:55:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 586206208 bytes 2024-09-16T12:55:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 618530816 bytes 2024-09-16T12:55:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 658797568 bytes 2024-09-16T12:55:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 706014208 bytes 2024-09-16T12:55:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 779538432 bytes 2024-09-16T12:55:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 808298496 bytes 2024-09-16T12:55:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 864416768 bytes 2024-09-16T12:55:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 905153536 bytes 2024-09-16T12:55:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 916781056 bytes 2024-09-16T12:55:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 990997504 bytes 2024-09-16T12:55:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1038978048 bytes 2024-09-16T12:55:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1039608832 bytes 2024-09-16T12:55:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1085301760 bytes 2024-09-16T12:55:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1108656128 bytes 2024-09-16T12:55:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1150745600 bytes 2024-09-16T12:55:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1206810624 bytes 2024-09-16T12:55:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1277641728 bytes 2024-09-16T12:55:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1342300160 bytes 2024-09-16T12:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1403514880 bytes 2024-09-16T12:55:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1422768128 bytes 2024-09-16T12:55:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1464256512 bytes 2024-09-16T12:55:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1489067008 bytes 2024-09-16T12:55:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1555349504 bytes 2024-09-16T12:55:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1597770752 bytes 2024-09-16T12:55:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1617844224 bytes 2024-09-16T12:55:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1675854848 bytes 2024-09-16T12:55:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1720024064 bytes 2024-09-16T12:55:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1770484736 bytes 2024-09-16T12:55:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1795269632 bytes 2024-09-16T12:55:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1862958080 bytes 2024-09-16T12:55:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1891180544 bytes 2024-09-16T12:55:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1921825792 bytes 2024-09-16T12:55:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1952505856 bytes 2024-09-16T12:55:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1983923200 bytes 2024-09-16T12:55:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2070574080 bytes 2024-09-16T12:55:08.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.977278080251548) > weight (0.25) 2024-09-16T12:55:08.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4217932764058542) > weight (0.25) 2024-09-16T12:55:08.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34552649421386716) > weight (0.25) 2024-09-16T12:55:08.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5907886596185852) > weight (0.25) 2024-09-16T12:55:08.846 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-16T12:55:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2143945728 bytes 2024-09-16T12:55:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2149171200 bytes 2024-09-16T12:55:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2177333248 bytes 2024-09-16T12:55:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2191593472 bytes 2024-09-16T12:55:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2239497216 bytes 2024-09-16T12:55:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2315876352 bytes 2024-09-16T12:55:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2377213952 bytes 2024-09-16T12:55:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2420294656 bytes 2024-09-16T12:55:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2494023680 bytes 2024-09-16T12:55:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2531384320 bytes 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:55:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-09-16T12:55:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:55:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-16T12:55:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T12:55:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T12:55:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T12:55:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12b297a64a2f 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12b297a64a2f 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:10 2024 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:10 2024 2024-09-16T12:55:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:10 2024 2024-09-16T12:55:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 435d688a-930a-49fa-824b-5ef9566d2d8e 2024-09-16T12:55:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T12:55:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T12:55:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 59038720 bytes 2024-09-16T12:55:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 121554944 bytes 2024-09-16T12:55:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 150532096 bytes 2024-09-16T12:55:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 194153472 bytes 2024-09-16T12:55:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 209320960 bytes 2024-09-16T12:55:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 276019200 bytes 2024-09-16T12:55:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 314545152 bytes 2024-09-16T12:55:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 338617344 bytes 2024-09-16T12:55:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 363184128 bytes 2024-09-16T12:55:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 407620608 bytes 2024-09-16T12:55:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 427583488 bytes 2024-09-16T12:55:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 471741440 bytes 2024-09-16T12:55:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 531555328 bytes 2024-09-16T12:55:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 553900032 bytes 2024-09-16T12:55:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 645632000 bytes 2024-09-16T12:55:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 665839616 bytes 2024-09-16T12:55:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 757617664 bytes 2024-09-16T12:55:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 822598656 bytes 2024-09-16T12:55:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 889812992 bytes 2024-09-16T12:55:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 890424320 bytes 2024-09-16T12:55:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 920071168 bytes 2024-09-16T12:55:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 974777344 bytes 2024-09-16T12:55:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 990458880 bytes 2024-09-16T12:55:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1033571328 bytes 2024-09-16T12:55:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1037816832 bytes 2024-09-16T12:55:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1075675136 bytes 2024-09-16T12:55:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1151039488 bytes 2024-09-16T12:55:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1181486080 bytes 2024-09-16T12:55:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1211340800 bytes 2024-09-16T12:55:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1250877440 bytes 2024-09-16T12:55:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1285268480 bytes 2024-09-16T12:55:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1325908992 bytes 2024-09-16T12:55:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1341394944 bytes 2024-09-16T12:55:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1391532032 bytes 2024-09-16T12:55:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1447711744 bytes 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T12:55:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:55:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:55:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:55:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-09-16T12:55:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:55:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-16T12:55:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T12:55:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T12:55:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T12:55:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-16T12:55:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12bedc7eeb79 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12bedc7eeb79 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:17 2024 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:17 2024 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:17 2024 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 44d2ade4-d0ac-4f1d-a249-9712f1c710b8 2024-09-16T12:55:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T12:55:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T12:55:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T12:55:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39400448 bytes 2024-09-16T12:55:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 60735488 bytes 2024-09-16T12:55:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 71203840 bytes 2024-09-16T12:55:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 137630720 bytes 2024-09-16T12:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 215141376 bytes 2024-09-16T12:55:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 306853888 bytes 2024-09-16T12:55:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 319015936 bytes 2024-09-16T12:55:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 390742016 bytes 2024-09-16T12:55:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 411491328 bytes 2024-09-16T12:55:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 438386688 bytes 2024-09-16T12:55:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 467777536 bytes 2024-09-16T12:55:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 496940032 bytes 2024-09-16T12:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 528611328 bytes 2024-09-16T12:55:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 643291136 bytes 2024-09-16T12:55:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 643920896 bytes 2024-09-16T12:55:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 672654336 bytes 2024-09-16T12:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 737039360 bytes 2024-09-16T12:55:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 819861504 bytes 2024-09-16T12:55:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 866525184 bytes 2024-09-16T12:55:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 932551680 bytes 2024-09-16T12:55:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 964501504 bytes 2024-09-16T12:55:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1003012096 bytes 2024-09-16T12:55:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1028762624 bytes 2024-09-16T12:55:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1104691200 bytes 2024-09-16T12:55:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1114815488 bytes 2024-09-16T12:55:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1137512448 bytes 2024-09-16T12:55:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1196741632 bytes 2024-09-16T12:55:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1253072896 bytes 2024-09-16T12:55:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1284171776 bytes 2024-09-16T12:55:23.761 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3268626380562947) > weight (0.25) 2024-09-16T12:55:23.761 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9339011786269951) > weight (0.25) 2024-09-16T12:55:23.761 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T12:55:23.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T12:55:23.762 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T12:55:23.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T12:55:23.761+0000 7faf1a3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 41113) UID: 1000 2024-09-16T12:55:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1361569792 bytes 2024-09-16T12:55:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1402249216 bytes 2024-09-16T12:55:24.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:55:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1445787648 bytes 2024-09-16T12:55:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1539186688 bytes 2024-09-16T12:55:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1583109120 bytes 2024-09-16T12:55:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1611510784 bytes 2024-09-16T12:55:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1654396928 bytes 2024-09-16T12:55:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1706147840 bytes 2024-09-16T12:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1756552192 bytes 2024-09-16T12:55:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1788294144 bytes 2024-09-16T12:55:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1808967680 bytes 2024-09-16T12:55:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1887139840 bytes 2024-09-16T12:55:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1954088960 bytes 2024-09-16T12:55:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1991086080 bytes 2024-09-16T12:55:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2030477312 bytes 2024-09-16T12:55:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2075494400 bytes 2024-09-16T12:55:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2093091840 bytes 2024-09-16T12:55:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2116346880 bytes 2024-09-16T12:55:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2125716480 bytes 2024-09-16T12:55:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2196246528 bytes 2024-09-16T12:55:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2199264256 bytes 2024-09-16T12:55:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2221617152 bytes 2024-09-16T12:55:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2313078784 bytes 2024-09-16T12:55:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2325860352 bytes 2024-09-16T12:55:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2336784384 bytes 2024-09-16T12:55:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2364184576 bytes 2024-09-16T12:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2401950720 bytes 2024-09-16T12:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2436060160 bytes 2024-09-16T12:55:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2500813824 bytes 2024-09-16T12:55:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2524514304 bytes 2024-09-16T12:55:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2573640704 bytes 2024-09-16T12:55:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2607289344 bytes 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-16T12:55:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-09-16T12:55:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:55:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-16T12:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T12:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T12:55:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T12:55:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-16T12:55:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12c92960e1f8 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12c92960e1f8 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:28 2024 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:28 2024 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:28 2024 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: e7069a5d-0e13-4d1b-976c-04e9409a6d09 2024-09-16T12:55:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T12:55:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T12:55:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 31620096 bytes 2024-09-16T12:55:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 95571968 bytes 2024-09-16T12:55:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 110544896 bytes 2024-09-16T12:55:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 111169536 bytes 2024-09-16T12:55:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 156427264 bytes 2024-09-16T12:55:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 188168192 bytes 2024-09-16T12:55:30.764 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T12:55:30.764 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:55:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:55:30.765 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T12:55:30.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T12:55:30.765 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T12:55:30.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T12:55:30.768 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-16T12:55:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 253361152 bytes 2024-09-16T12:55:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 320108544 bytes 2024-09-16T12:55:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 389478400 bytes 2024-09-16T12:55:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 460336128 bytes 2024-09-16T12:55:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 503872512 bytes 2024-09-16T12:55:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 516398080 bytes 2024-09-16T12:55:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 569862144 bytes 2024-09-16T12:55:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 603405312 bytes 2024-09-16T12:55:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 646531072 bytes 2024-09-16T12:55:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 718194688 bytes 2024-09-16T12:55:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 718798848 bytes 2024-09-16T12:55:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 738388992 bytes 2024-09-16T12:55:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 782256128 bytes 2024-09-16T12:55:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 860283904 bytes 2024-09-16T12:55:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 922687488 bytes 2024-09-16T12:55:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 962505728 bytes 2024-09-16T12:55:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1036738560 bytes 2024-09-16T12:55:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1071496192 bytes 2024-09-16T12:55:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1131307008 bytes 2024-09-16T12:55:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1162826752 bytes 2024-09-16T12:55:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1197059072 bytes 2024-09-16T12:55:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1203432448 bytes 2024-09-16T12:55:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1230897152 bytes 2024-09-16T12:55:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1251339264 bytes 2024-09-16T12:55:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1322845184 bytes 2024-09-16T12:55:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1368808448 bytes 2024-09-16T12:55:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1438641152 bytes 2024-09-16T12:55:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1474603008 bytes 2024-09-16T12:55:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1546161152 bytes 2024-09-16T12:55:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1592116224 bytes 2024-09-16T12:55:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1664474112 bytes 2024-09-16T12:55:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1764157440 bytes 2024-09-16T12:55:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1796846592 bytes 2024-09-16T12:55:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1874322432 bytes 2024-09-16T12:55:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1888465920 bytes 2024-09-16T12:55:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1922629632 bytes 2024-09-16T12:55:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1992427520 bytes 2024-09-16T12:55:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2059800576 bytes 2024-09-16T12:55:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2114269184 bytes 2024-09-16T12:55:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2163878912 bytes 2024-09-16T12:55:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2231190528 bytes 2024-09-16T12:55:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2298589184 bytes 2024-09-16T12:55:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2428208128 bytes 2024-09-16T12:55:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2472446976 bytes 2024-09-16T12:55:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2511243264 bytes 2024-09-16T12:55:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2557673472 bytes 2024-09-16T12:55:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2570883072 bytes 2024-09-16T12:55:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2627558400 bytes 2024-09-16T12:55:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2689223680 bytes 2024-09-16T12:55:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-16T12:55:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:55:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-09-16T12:55:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:55:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-16T12:55:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T12:55:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T12:55:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12df6ac4f5a9 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12df6ac4f5a9 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:39 2024 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:39 2024 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:39 2024 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 470fe0b0-f9fc-45ce-91f1-79e5c3c5fc0e 2024-09-16T12:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T12:55:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T12:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-16T12:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 92059648 bytes 2024-09-16T12:55:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 140488704 bytes 2024-09-16T12:55:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 151565312 bytes 2024-09-16T12:55:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 197377024 bytes 2024-09-16T12:55:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 245767168 bytes 2024-09-16T12:55:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 263179264 bytes 2024-09-16T12:55:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 291799040 bytes 2024-09-16T12:55:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 331546624 bytes 2024-09-16T12:55:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 396069888 bytes 2024-09-16T12:55:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 396681216 bytes 2024-09-16T12:55:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 417501184 bytes 2024-09-16T12:55:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 443763712 bytes 2024-09-16T12:55:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 493212672 bytes 2024-09-16T12:55:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 572396544 bytes 2024-09-16T12:55:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 597058560 bytes 2024-09-16T12:55:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 651016192 bytes 2024-09-16T12:55:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 744914944 bytes 2024-09-16T12:55:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 786979840 bytes 2024-09-16T12:55:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 809821184 bytes 2024-09-16T12:55:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 854590464 bytes 2024-09-16T12:55:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 905907200 bytes 2024-09-16T12:55:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 961897472 bytes 2024-09-16T12:55:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 996345856 bytes 2024-09-16T12:55:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1047938048 bytes 2024-09-16T12:55:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1048556544 bytes 2024-09-16T12:55:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1100348416 bytes 2024-09-16T12:55:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1110193152 bytes 2024-09-16T12:55:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1149546496 bytes 2024-09-16T12:55:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1220474880 bytes 2024-09-16T12:55:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1234827264 bytes 2024-09-16T12:55:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1311526912 bytes 2024-09-16T12:55:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1356567552 bytes 2024-09-16T12:55:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1427034112 bytes 2024-09-16T12:55:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1477068800 bytes 2024-09-16T12:55:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1511318528 bytes 2024-09-16T12:55:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1530161152 bytes 2024-09-16T12:55:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1602958336 bytes 2024-09-16T12:55:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1610540032 bytes 2024-09-16T12:55:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1680392192 bytes 2024-09-16T12:55:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1700667392 bytes 2024-09-16T12:55:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1778267136 bytes 2024-09-16T12:55:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1815093248 bytes 2024-09-16T12:55:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1847351296 bytes 2024-09-16T12:55:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1901752320 bytes 2024-09-16T12:55:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1990534144 bytes 2024-09-16T12:55:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2032984064 bytes 2024-09-16T12:55:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2057433088 bytes 2024-09-16T12:55:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2083719168 bytes 2024-09-16T12:55:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2118815744 bytes 2024-09-16T12:55:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2151021568 bytes 2024-09-16T12:55:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2182754304 bytes 2024-09-16T12:55:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2237231104 bytes 2024-09-16T12:55:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2262549504 bytes 2024-09-16T12:55:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2303511552 bytes 2024-09-16T12:55:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2366234624 bytes 2024-09-16T12:55:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2417100800 bytes 2024-09-16T12:55:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2439701504 bytes 2024-09-16T12:55:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2471003136 bytes 2024-09-16T12:55:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2494936064 bytes 2024-09-16T12:55:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2592041984 bytes 2024-09-16T12:55:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2620316672 bytes 2024-09-16T12:55:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2679312384 bytes 2024-09-16T12:55:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2698345472 bytes 2024-09-16T12:55:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2762847232 bytes 2024-09-16T12:55:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2800237568 bytes 2024-09-16T12:55:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T12:55:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:55:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:55:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:55:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:55:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-09-16T12:55:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:55:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T12:55:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:55:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12ebe3bb2e97 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12ebe3bb2e97 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:55:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:52 2024 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:52 2024 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:52 2024 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: aa2d39b0-ecda-4092-b3c6-43b942830859 2024-09-16T12:55:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T12:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T12:55:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:55:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-16T12:55:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:55:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:55:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 47670272 bytes 2024-09-16T12:55:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 109795328 bytes 2024-09-16T12:55:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 153524224 bytes 2024-09-16T12:55:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 188741632 bytes 2024-09-16T12:55:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 210092032 bytes 2024-09-16T12:55:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 217920512 bytes 2024-09-16T12:55:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 238042112 bytes 2024-09-16T12:55:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 264881152 bytes 2024-09-16T12:55:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 344605696 bytes 2024-09-16T12:55:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 360216576 bytes 2024-09-16T12:55:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 454515712 bytes 2024-09-16T12:55:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 482813952 bytes 2024-09-16T12:55:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 493845504 bytes 2024-09-16T12:55:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 528976896 bytes 2024-09-16T12:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 552826880 bytes 2024-09-16T12:55:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 608372736 bytes 2024-09-16T12:55:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 636068864 bytes 2024-09-16T12:55:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 688607232 bytes 2024-09-16T12:55:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 724729856 bytes 2024-09-16T12:55:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 830274560 bytes 2024-09-16T12:55:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 864950272 bytes 2024-09-16T12:55:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 913358848 bytes 2024-09-16T12:55:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 984192000 bytes 2024-09-16T12:55:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 999658496 bytes 2024-09-16T12:55:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1053798400 bytes 2024-09-16T12:55:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1069796352 bytes 2024-09-16T12:55:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1112266752 bytes 2024-09-16T12:55:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1112892416 bytes 2024-09-16T12:55:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1165330432 bytes 2024-09-16T12:55:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1197760512 bytes 2024-09-16T12:55:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1233737728 bytes 2024-09-16T12:55:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1305279488 bytes 2024-09-16T12:55:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1311439872 bytes 2024-09-16T12:55:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1345851392 bytes 2024-09-16T12:55:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1380984832 bytes 2024-09-16T12:55:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1508282368 bytes 2024-09-16T12:55:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1537874944 bytes 2024-09-16T12:55:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1547138048 bytes 2024-09-16T12:55:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1625089024 bytes 2024-09-16T12:55:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1628996608 bytes 2024-09-16T12:55:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1648777216 bytes 2024-09-16T12:55:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1692304384 bytes 2024-09-16T12:55:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1704174592 bytes 2024-09-16T12:55:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1756867584 bytes 2024-09-16T12:56:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1801732096 bytes 2024-09-16T12:56:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1833422848 bytes 2024-09-16T12:56:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1859801088 bytes 2024-09-16T12:56:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1868128256 bytes 2024-09-16T12:56:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1904850944 bytes 2024-09-16T12:56:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1968388096 bytes 2024-09-16T12:56:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2027022336 bytes 2024-09-16T12:56:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2035266560 bytes 2024-09-16T12:56:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2102752256 bytes 2024-09-16T12:56:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2141399040 bytes 2024-09-16T12:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2235939840 bytes 2024-09-16T12:56:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2261926912 bytes 2024-09-16T12:56:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2277422080 bytes 2024-09-16T12:56:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2327325696 bytes 2024-09-16T12:56:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2397100032 bytes 2024-09-16T12:56:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2476666880 bytes 2024-09-16T12:56:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2533015552 bytes 2024-09-16T12:56:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2588587008 bytes 2024-09-16T12:56:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2604081152 bytes 2024-09-16T12:56:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2633496576 bytes 2024-09-16T12:56:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2640016384 bytes 2024-09-16T12:56:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2640648192 bytes 2024-09-16T12:56:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2641292288 bytes 2024-09-16T12:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2710948864 bytes 2024-09-16T12:56:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2738115584 bytes 2024-09-16T12:56:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2781157376 bytes 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-16T12:56:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T12:56:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T12:56:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-16T12:56:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12f747f94bc0 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12f747f94bc0 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:04 2024 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:04 2024 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:04 2024 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a9b282ed-f577-495d-bb2c-a0ac91558148 2024-09-16T12:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T12:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-16T12:56:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 46851072 bytes 2024-09-16T12:56:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 99362816 bytes 2024-09-16T12:56:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 132580352 bytes 2024-09-16T12:56:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 194537472 bytes 2024-09-16T12:56:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 215958528 bytes 2024-09-16T12:56:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 346411008 bytes 2024-09-16T12:56:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 440786944 bytes 2024-09-16T12:56:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 489499648 bytes 2024-09-16T12:56:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 498148352 bytes 2024-09-16T12:56:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 580472832 bytes 2024-09-16T12:56:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 603311104 bytes 2024-09-16T12:56:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 649354240 bytes 2024-09-16T12:56:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 729919488 bytes 2024-09-16T12:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 740135936 bytes 2024-09-16T12:56:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 760872960 bytes 2024-09-16T12:56:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 887471104 bytes 2024-09-16T12:56:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 933336064 bytes 2024-09-16T12:56:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 961294336 bytes 2024-09-16T12:56:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 971395072 bytes 2024-09-16T12:56:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1050805248 bytes 2024-09-16T12:56:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1162419200 bytes 2024-09-16T12:56:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1169639424 bytes 2024-09-16T12:56:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1193193472 bytes 2024-09-16T12:56:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1222774784 bytes 2024-09-16T12:56:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1327348736 bytes 2024-09-16T12:56:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1368000512 bytes 2024-09-16T12:56:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1402021888 bytes 2024-09-16T12:56:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1435828224 bytes 2024-09-16T12:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1489415168 bytes 2024-09-16T12:56:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1565440000 bytes 2024-09-16T12:56:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1623763968 bytes 2024-09-16T12:56:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1675640832 bytes 2024-09-16T12:56:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1722383360 bytes 2024-09-16T12:56:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1778997248 bytes 2024-09-16T12:56:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1812024320 bytes 2024-09-16T12:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1909969920 bytes 2024-09-16T12:56:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1977372672 bytes 2024-09-16T12:56:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 2017040384 bytes 2024-09-16T12:56:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 2055832576 bytes 2024-09-16T12:56:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2135604224 bytes 2024-09-16T12:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2158286848 bytes 2024-09-16T12:56:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2249947136 bytes 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-16T12:56:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-09-16T12:56:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-16T12:56:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T12:56:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-16T12:56:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-16T12:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_11': 2024-09-16T12:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 130357c96780 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.130357c96780 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:13 2024 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:13 2024 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:13 2024 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 000b0e94-a5bd-4706-9afd-99ff1b8bf462 2024-09-16T12:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T12:56:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T12:56:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 46149632 bytes 2024-09-16T12:56:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 68278272 bytes 2024-09-16T12:56:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 94687232 bytes 2024-09-16T12:56:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 117902336 bytes 2024-09-16T12:56:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 200840192 bytes 2024-09-16T12:56:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 292838400 bytes 2024-09-16T12:56:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 293476352 bytes 2024-09-16T12:56:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 345988096 bytes 2024-09-16T12:56:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 411420672 bytes 2024-09-16T12:56:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 417254400 bytes 2024-09-16T12:56:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 467254272 bytes 2024-09-16T12:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 518875136 bytes 2024-09-16T12:56:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 591818752 bytes 2024-09-16T12:56:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 633837568 bytes 2024-09-16T12:56:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 637732864 bytes 2024-09-16T12:56:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 662156288 bytes 2024-09-16T12:56:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 717676544 bytes 2024-09-16T12:56:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 791809024 bytes 2024-09-16T12:56:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 842671104 bytes 2024-09-16T12:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 952485888 bytes 2024-09-16T12:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1021406208 bytes 2024-09-16T12:56:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1066891264 bytes 2024-09-16T12:56:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1120303104 bytes 2024-09-16T12:56:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1172156416 bytes 2024-09-16T12:56:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1185148928 bytes 2024-09-16T12:56:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1236853760 bytes 2024-09-16T12:56:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1339192320 bytes 2024-09-16T12:56:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1388609536 bytes 2024-09-16T12:56:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1402347520 bytes 2024-09-16T12:56:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1439395840 bytes 2024-09-16T12:56:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1494069248 bytes 2024-09-16T12:56:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1526321152 bytes 2024-09-16T12:56:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1556112384 bytes 2024-09-16T12:56:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1581684736 bytes 2024-09-16T12:56:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1622142976 bytes 2024-09-16T12:56:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1662711808 bytes 2024-09-16T12:56:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1677450240 bytes 2024-09-16T12:56:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1695951872 bytes 2024-09-16T12:56:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1725571072 bytes 2024-09-16T12:56:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1753227264 bytes 2024-09-16T12:56:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1821670400 bytes 2024-09-16T12:56:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1939996672 bytes 2024-09-16T12:56:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1989944320 bytes 2024-09-16T12:56:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2034810880 bytes 2024-09-16T12:56:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2129342464 bytes 2024-09-16T12:56:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2149792768 bytes 2024-09-16T12:56:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2153568256 bytes 2024-09-16T12:56:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2203245568 bytes 2024-09-16T12:56:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2238490624 bytes 2024-09-16T12:56:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2275033088 bytes 2024-09-16T12:56:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2372121600 bytes 2024-09-16T12:56:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2437463040 bytes 2024-09-16T12:56:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2529947648 bytes 2024-09-16T12:56:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2596421632 bytes 2024-09-16T12:56:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2635130880 bytes 2024-09-16T12:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2651723776 bytes 2024-09-16T12:56:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2677128192 bytes 2024-09-16T12:56:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2788115456 bytes 2024-09-16T12:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2856074240 bytes 2024-09-16T12:56:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2880777216 bytes 2024-09-16T12:56:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2931383296 bytes 2024-09-16T12:56:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 3004742656 bytes 2024-09-16T12:56:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3063549952 bytes 2024-09-16T12:56:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3083921408 bytes 2024-09-16T12:56:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3090114560 bytes 2024-09-16T12:56:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3149026304 bytes 2024-09-16T12:56:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3197449216 bytes 2024-09-16T12:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3223070720 bytes 2024-09-16T12:56:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3270307840 bytes 2024-09-16T12:56:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3270917120 bytes 2024-09-16T12:56:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3295941632 bytes 2024-09-16T12:56:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3296544768 bytes 2024-09-16T12:56:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3326120960 bytes 2024-09-16T12:56:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3364790272 bytes 2024-09-16T12:56:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-09-16T12:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-16T12:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T12:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-16T12:56:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-16T12:56:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_12': 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 130f39a11944 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.130f39a11944 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:27 2024 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:27 2024 2024-09-16T12:56:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:27 2024 2024-09-16T12:56:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 42c7f0d4-ea22-4303-82c7-5d306ec75295 2024-09-16T12:56:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-16T12:56:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T12:56:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T12:56:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 73027584 bytes 2024-09-16T12:56:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 117366784 bytes 2024-09-16T12:56:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 122635264 bytes 2024-09-16T12:56:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 166664192 bytes 2024-09-16T12:56:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 238096384 bytes 2024-09-16T12:56:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 258867200 bytes 2024-09-16T12:56:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 339051520 bytes 2024-09-16T12:56:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 360344576 bytes 2024-09-16T12:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 435145728 bytes 2024-09-16T12:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 550084608 bytes 2024-09-16T12:56:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 592187392 bytes 2024-09-16T12:56:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 653425664 bytes 2024-09-16T12:56:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 684529664 bytes 2024-09-16T12:56:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 705131520 bytes 2024-09-16T12:56:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 780716032 bytes 2024-09-16T12:56:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 831042560 bytes 2024-09-16T12:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 851575808 bytes 2024-09-16T12:56:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 890243072 bytes 2024-09-16T12:56:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 921529344 bytes 2024-09-16T12:56:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 966942720 bytes 2024-09-16T12:56:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 980276224 bytes 2024-09-16T12:56:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 980913152 bytes 2024-09-16T12:56:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 999779328 bytes 2024-09-16T12:56:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1050371072 bytes 2024-09-16T12:56:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1124096000 bytes 2024-09-16T12:56:32.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9418466357058882) > weight (0.25) 2024-09-16T12:56:32.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3454983722164292) > weight (0.25) 2024-09-16T12:56:32.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7310263086992786) > weight (0.25) 2024-09-16T12:56:32.847 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7646991664266238) > weight (0.25) 2024-09-16T12:56:32.847 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T12:56:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1173000192 bytes 2024-09-16T12:56:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1184667648 bytes 2024-09-16T12:56:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1316153344 bytes 2024-09-16T12:56:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1337777152 bytes 2024-09-16T12:56:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1355985920 bytes 2024-09-16T12:56:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1365538816 bytes 2024-09-16T12:56:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1444114432 bytes 2024-09-16T12:56:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1460513792 bytes 2024-09-16T12:56:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1504291840 bytes 2024-09-16T12:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1506871296 bytes 2024-09-16T12:56:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1517221888 bytes 2024-09-16T12:56:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-16T12:56:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-16T12:56:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_13': 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 131b37fcaf92 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.131b37fcaf92 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:34 2024 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:34 2024 2024-09-16T12:56:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:34 2024 2024-09-16T12:56:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 451fa1a3-9713-4bdb-894c-abccb64adbc2 2024-09-16T12:56:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-16T12:56:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T12:56:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T12:56:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 62691328 bytes 2024-09-16T12:56:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 87867392 bytes 2024-09-16T12:56:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 108376064 bytes 2024-09-16T12:56:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 150170624 bytes 2024-09-16T12:56:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 188395520 bytes 2024-09-16T12:56:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 255518720 bytes 2024-09-16T12:56:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 281254912 bytes 2024-09-16T12:56:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 285916160 bytes 2024-09-16T12:56:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 324453376 bytes 2024-09-16T12:56:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 378479616 bytes 2024-09-16T12:56:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 433076224 bytes 2024-09-16T12:56:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 466219008 bytes 2024-09-16T12:56:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 513788928 bytes 2024-09-16T12:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 547141632 bytes 2024-09-16T12:56:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 594739200 bytes 2024-09-16T12:56:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 637811712 bytes 2024-09-16T12:56:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 666706944 bytes 2024-09-16T12:56:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 708036608 bytes 2024-09-16T12:56:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 778962944 bytes 2024-09-16T12:56:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 845156352 bytes 2024-09-16T12:56:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 880122880 bytes 2024-09-16T12:56:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 904221696 bytes 2024-09-16T12:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 904815616 bytes 2024-09-16T12:56:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 954126336 bytes 2024-09-16T12:56:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 999406592 bytes 2024-09-16T12:56:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1050158080 bytes 2024-09-16T12:56:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1078649856 bytes 2024-09-16T12:56:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1098202112 bytes 2024-09-16T12:56:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1164836864 bytes 2024-09-16T12:56:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1252853760 bytes 2024-09-16T12:56:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1278274560 bytes 2024-09-16T12:56:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1303432192 bytes 2024-09-16T12:56:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1338098688 bytes 2024-09-16T12:56:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1387771904 bytes 2024-09-16T12:56:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1436725248 bytes 2024-09-16T12:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-16T12:56:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-16T12:56:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_14': 2024-09-16T12:56:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1327a78dd5c5 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1327a78dd5c5 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:41 2024 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:41 2024 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:41 2024 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f3868c68-fe40-4444-8720-528d92e98de6 2024-09-16T12:56:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-16T12:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T12:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:56:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 122918912 bytes 2024-09-16T12:56:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 143369216 bytes 2024-09-16T12:56:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 186204160 bytes 2024-09-16T12:56:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 264781824 bytes 2024-09-16T12:56:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 318558208 bytes 2024-09-16T12:56:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 362836992 bytes 2024-09-16T12:56:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 378973184 bytes 2024-09-16T12:56:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 402321408 bytes 2024-09-16T12:56:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 474458112 bytes 2024-09-16T12:56:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 475072512 bytes 2024-09-16T12:56:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 514407424 bytes 2024-09-16T12:56:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 566245376 bytes 2024-09-16T12:56:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 575644672 bytes 2024-09-16T12:56:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 599184384 bytes 2024-09-16T12:56:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 599805952 bytes 2024-09-16T12:56:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 642567168 bytes 2024-09-16T12:56:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 745201664 bytes 2024-09-16T12:56:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 813369344 bytes 2024-09-16T12:56:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 849211392 bytes 2024-09-16T12:56:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 895934464 bytes 2024-09-16T12:56:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 907126784 bytes 2024-09-16T12:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 987357184 bytes 2024-09-16T12:56:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1089494016 bytes 2024-09-16T12:56:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1104657408 bytes 2024-09-16T12:56:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1134599168 bytes 2024-09-16T12:56:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1179681792 bytes 2024-09-16T12:56:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1185987584 bytes 2024-09-16T12:56:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1263948800 bytes 2024-09-16T12:56:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1285855232 bytes 2024-09-16T12:56:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1364214784 bytes 2024-09-16T12:56:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1405174784 bytes 2024-09-16T12:56:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1448255488 bytes 2024-09-16T12:56:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1482097664 bytes 2024-09-16T12:56:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1532580864 bytes 2024-09-16T12:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1564160000 bytes 2024-09-16T12:56:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1665915904 bytes 2024-09-16T12:56:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1716460544 bytes 2024-09-16T12:56:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1754042368 bytes 2024-09-16T12:56:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1788188672 bytes 2024-09-16T12:56:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:56:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:56:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:56:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:56:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-16T12:56:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:56:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_15': 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1333737a725 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1333737a725 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:49 2024 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:49 2024 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:49 2024 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f65d50c0-9e44-4158-8c3d-91b35d21fe9a 2024-09-16T12:56:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T12:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-16T12:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:56:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 34697216 bytes 2024-09-16T12:56:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 95532032 bytes 2024-09-16T12:56:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 162655232 bytes 2024-09-16T12:56:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 224694272 bytes 2024-09-16T12:56:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 246870016 bytes 2024-09-16T12:56:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 262239232 bytes 2024-09-16T12:56:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 298743808 bytes 2024-09-16T12:56:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 373753856 bytes 2024-09-16T12:56:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 405671936 bytes 2024-09-16T12:56:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 413500416 bytes 2024-09-16T12:56:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 474456064 bytes 2024-09-16T12:56:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 547215360 bytes 2024-09-16T12:56:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 592335872 bytes 2024-09-16T12:56:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 594813952 bytes 2024-09-16T12:56:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 648987648 bytes 2024-09-16T12:56:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 649588736 bytes 2024-09-16T12:56:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 672816128 bytes 2024-09-16T12:56:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 673460224 bytes 2024-09-16T12:56:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 703248384 bytes 2024-09-16T12:56:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 724133888 bytes 2024-09-16T12:56:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 780101632 bytes 2024-09-16T12:56:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 807394304 bytes 2024-09-16T12:56:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 854775808 bytes 2024-09-16T12:56:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 928110592 bytes 2024-09-16T12:56:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1028041728 bytes 2024-09-16T12:56:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1031894016 bytes 2024-09-16T12:56:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1032499200 bytes 2024-09-16T12:56:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1048492032 bytes 2024-09-16T12:56:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1112705024 bytes 2024-09-16T12:56:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1143146496 bytes 2024-09-16T12:56:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1154668544 bytes 2024-09-16T12:56:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1209051136 bytes 2024-09-16T12:56:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1209680896 bytes 2024-09-16T12:56:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1237680128 bytes 2024-09-16T12:56:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1288924160 bytes 2024-09-16T12:56:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1367564288 bytes 2024-09-16T12:56:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1402873856 bytes 2024-09-16T12:56:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1443484672 bytes 2024-09-16T12:56:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1444237312 bytes 2024-09-16T12:56:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1502410752 bytes 2024-09-16T12:56:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1586370560 bytes 2024-09-16T12:56:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1627102208 bytes 2024-09-16T12:56:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1631377408 bytes 2024-09-16T12:56:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1691023360 bytes 2024-09-16T12:56:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1755826176 bytes 2024-09-16T12:56:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1803701248 bytes 2024-09-16T12:56:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1808633856 bytes 2024-09-16T12:56:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1834817536 bytes 2024-09-16T12:56:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1870108672 bytes 2024-09-16T12:56:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1875858432 bytes 2024-09-16T12:56:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1876664320 bytes 2024-09-16T12:56:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1980801024 bytes 2024-09-16T12:56:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2007318528 bytes 2024-09-16T12:56:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2053131264 bytes 2024-09-16T12:56:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2104344576 bytes 2024-09-16T12:56:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2143214592 bytes 2024-09-16T12:56:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2157619200 bytes 2024-09-16T12:56:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2227532800 bytes 2024-09-16T12:56:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2228440064 bytes 2024-09-16T12:56:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2258194432 bytes 2024-09-16T12:56:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2327065600 bytes 2024-09-16T12:56:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2356503552 bytes 2024-09-16T12:56:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2405605376 bytes 2024-09-16T12:56:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2406218752 bytes 2024-09-16T12:57:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2454196224 bytes 2024-09-16T12:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2533796864 bytes 2024-09-16T12:57:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2551775232 bytes 2024-09-16T12:57:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2608690176 bytes 2024-09-16T12:57:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2632407040 bytes 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-16T12:57:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-16T12:57:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:57:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_16': 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 133f340f6d4b 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.133f340f6d4b 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:00 2024 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:00 2024 2024-09-16T12:57:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:00 2024 2024-09-16T12:57:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:57:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:57:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f77bbb21-fc88-4735-8cf0-73c5d5cc573f 2024-09-16T12:57:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T12:57:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:57:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T12:57:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:57:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:57:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 43097088 bytes 2024-09-16T12:57:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 159789056 bytes 2024-09-16T12:57:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 225729536 bytes 2024-09-16T12:57:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 239200256 bytes 2024-09-16T12:57:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 295356416 bytes 2024-09-16T12:57:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 402583552 bytes 2024-09-16T12:57:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 484393984 bytes 2024-09-16T12:57:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 512095232 bytes 2024-09-16T12:57:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 529866752 bytes 2024-09-16T12:57:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 571828224 bytes 2024-09-16T12:57:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 631176192 bytes 2024-09-16T12:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 702926848 bytes 2024-09-16T12:57:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 736169984 bytes 2024-09-16T12:57:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 772094976 bytes 2024-09-16T12:57:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 855785472 bytes 2024-09-16T12:57:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 950979584 bytes 2024-09-16T12:57:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 1052967936 bytes 2024-09-16T12:57:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 1078300672 bytes 2024-09-16T12:57:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1129124864 bytes 2024-09-16T12:57:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1197670400 bytes 2024-09-16T12:57:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1234627584 bytes 2024-09-16T12:57:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1274801152 bytes 2024-09-16T12:57:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1285528576 bytes 2024-09-16T12:57:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1298282496 bytes 2024-09-16T12:57:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1392379904 bytes 2024-09-16T12:57:06.769 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4963474162817202) > weight (0.25) 2024-09-16T12:57:06.769 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3314827529971516) > weight (0.25) 2024-09-16T12:57:06.769 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T12:57:06.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T12:57:06.770 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T12:57:06.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T12:57:06.768+0000 7f4cf07f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 47368) UID: 1000 2024-09-16T12:57:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1396987904 bytes 2024-09-16T12:57:06.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1429164032 bytes 2024-09-16T12:57:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1555977216 bytes 2024-09-16T12:57:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1611219968 bytes 2024-09-16T12:57:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1677186048 bytes 2024-09-16T12:57:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1720342528 bytes 2024-09-16T12:57:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1748774912 bytes 2024-09-16T12:57:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1787276288 bytes 2024-09-16T12:57:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1829890048 bytes 2024-09-16T12:57:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1857045504 bytes 2024-09-16T12:57:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1882877952 bytes 2024-09-16T12:57:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1892251648 bytes 2024-09-16T12:57:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1980581888 bytes 2024-09-16T12:57:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1989153792 bytes 2024-09-16T12:57:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2017770496 bytes 2024-09-16T12:57:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2088112128 bytes 2024-09-16T12:57:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2127066112 bytes 2024-09-16T12:57:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2173467648 bytes 2024-09-16T12:57:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2229211136 bytes 2024-09-16T12:57:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2246031360 bytes 2024-09-16T12:57:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2260588544 bytes 2024-09-16T12:57:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2291664896 bytes 2024-09-16T12:57:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2304937984 bytes 2024-09-16T12:57:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2307983360 bytes 2024-09-16T12:57:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2342631424 bytes 2024-09-16T12:57:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2392506368 bytes 2024-09-16T12:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2450393088 bytes 2024-09-16T12:57:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2455193600 bytes 2024-09-16T12:57:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2494073856 bytes 2024-09-16T12:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2499494912 bytes 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2579221504 bytes 2024-09-16T12:57:11.771 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T12:57:11.771 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:57:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:57:11.772 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T12:57:11.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T12:57:11.772 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T12:57:11.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T12:57:11.775 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-16T12:57:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2620589056 bytes 2024-09-16T12:57:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2644788224 bytes 2024-09-16T12:57:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2702516224 bytes 2024-09-16T12:57:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2783418368 bytes 2024-09-16T12:57:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2883417088 bytes 2024-09-16T12:57:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2900415488 bytes 2024-09-16T12:57:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2901044224 bytes 2024-09-16T12:57:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2939533312 bytes 2024-09-16T12:57:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3018208256 bytes 2024-09-16T12:57:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3058239488 bytes 2024-09-16T12:57:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3122075648 bytes 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:57:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:57:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:57:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-09-16T12:57:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-16T12:57:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:57:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_17': 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1354671f7989 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1354671f7989 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:57:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:13 2024 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:13 2024 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:13 2024 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 105a819b-f484-4d1d-a16e-4f48810acd47 2024-09-16T12:57:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T12:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:57:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T12:57:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:57:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:57:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 18122752 bytes 2024-09-16T12:57:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 51844096 bytes 2024-09-16T12:57:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 105114624 bytes 2024-09-16T12:57:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 119556096 bytes 2024-09-16T12:57:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 164019200 bytes 2024-09-16T12:57:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 205669376 bytes 2024-09-16T12:57:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 264592384 bytes 2024-09-16T12:57:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 306403328 bytes 2024-09-16T12:57:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 384358400 bytes 2024-09-16T12:57:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 455265280 bytes 2024-09-16T12:57:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 525897728 bytes 2024-09-16T12:57:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 632464384 bytes 2024-09-16T12:57:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 673207296 bytes 2024-09-16T12:57:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 681587712 bytes 2024-09-16T12:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 686979072 bytes 2024-09-16T12:57:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 721015808 bytes 2024-09-16T12:57:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 760970240 bytes 2024-09-16T12:57:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 798224384 bytes 2024-09-16T12:57:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 830167040 bytes 2024-09-16T12:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 871775232 bytes 2024-09-16T12:57:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 927848448 bytes 2024-09-16T12:57:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 986459136 bytes 2024-09-16T12:57:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1004890112 bytes 2024-09-16T12:57:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1023815680 bytes 2024-09-16T12:57:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1074934784 bytes 2024-09-16T12:57:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1117108224 bytes 2024-09-16T12:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1189975040 bytes 2024-09-16T12:57:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1271714816 bytes 2024-09-16T12:57:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1331743744 bytes 2024-09-16T12:57:20.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1360093184 bytes 2024-09-16T12:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1372085248 bytes 2024-09-16T12:57:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1412958208 bytes 2024-09-16T12:57:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1423169536 bytes 2024-09-16T12:57:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1476012032 bytes 2024-09-16T12:57:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1545260032 bytes 2024-09-16T12:57:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1627415552 bytes 2024-09-16T12:57:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1649010688 bytes 2024-09-16T12:57:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1699749888 bytes 2024-09-16T12:57:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1755986944 bytes 2024-09-16T12:57:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1802811392 bytes 2024-09-16T12:57:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1871415296 bytes 2024-09-16T12:57:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1916748800 bytes 2024-09-16T12:57:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1967215616 bytes 2024-09-16T12:57:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1976650752 bytes 2024-09-16T12:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2007851008 bytes 2024-09-16T12:57:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2049681408 bytes 2024-09-16T12:57:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2052376576 bytes 2024-09-16T12:57:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2108785664 bytes 2024-09-16T12:57:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2171459584 bytes 2024-09-16T12:57:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2208701440 bytes 2024-09-16T12:57:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2231597056 bytes 2024-09-16T12:57:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2254586880 bytes 2024-09-16T12:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2341904384 bytes 2024-09-16T12:57:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2378219520 bytes 2024-09-16T12:57:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2403838976 bytes 2024-09-16T12:57:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2421511168 bytes 2024-09-16T12:57:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2478860288 bytes 2024-09-16T12:57:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2548964352 bytes 2024-09-16T12:57:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2568258560 bytes 2024-09-16T12:57:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2572207104 bytes 2024-09-16T12:57:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2652451840 bytes 2024-09-16T12:57:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2725484544 bytes 2024-09-16T12:57:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2761051136 bytes 2024-09-16T12:57:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2817530880 bytes 2024-09-16T12:57:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2848662528 bytes 2024-09-16T12:57:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2901440512 bytes 2024-09-16T12:57:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2989184000 bytes 2024-09-16T12:57:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3055989760 bytes 2024-09-16T12:57:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3109871616 bytes 2024-09-16T12:57:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3132450816 bytes 2024-09-16T12:57:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3190162432 bytes 2024-09-16T12:57:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3218491392 bytes 2024-09-16T12:57:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3278341120 bytes 2024-09-16T12:57:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3344161792 bytes 2024-09-16T12:57:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3405916160 bytes 2024-09-16T12:57:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3467447296 bytes 2024-09-16T12:57:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3474593792 bytes 2024-09-16T12:57:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3494146048 bytes 2024-09-16T12:57:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3499103232 bytes 2024-09-16T12:57:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3571681280 bytes 2024-09-16T12:57:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3634155520 bytes 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:57:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:57:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:57:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-09-16T12:57:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:57:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-16T12:57:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T12:57:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-16T12:57:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:57:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_18': 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 136060485119 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.136060485119 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:57:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:28 2024 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:28 2024 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:28 2024 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d03e92a5-c8e6-4505-aae2-e8c01ac781fc 2024-09-16T12:57:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:57:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:57:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-16T12:57:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T12:57:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:57:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T12:57:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:57:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:57:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 41729024 bytes 2024-09-16T12:57:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 108157952 bytes 2024-09-16T12:57:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 172218368 bytes 2024-09-16T12:57:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 176521216 bytes 2024-09-16T12:57:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 259489792 bytes 2024-09-16T12:57:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 337944576 bytes 2024-09-16T12:57:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 363373568 bytes 2024-09-16T12:57:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 405407744 bytes 2024-09-16T12:57:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 458652672 bytes 2024-09-16T12:57:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 505271296 bytes 2024-09-16T12:57:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 585795584 bytes 2024-09-16T12:57:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 677335040 bytes 2024-09-16T12:57:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 705875968 bytes 2024-09-16T12:57:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 747809792 bytes 2024-09-16T12:57:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 783934464 bytes 2024-09-16T12:57:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 806203392 bytes 2024-09-16T12:57:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 883513344 bytes 2024-09-16T12:57:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 928904192 bytes 2024-09-16T12:57:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 966331392 bytes 2024-09-16T12:57:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1009159168 bytes 2024-09-16T12:57:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1064751104 bytes 2024-09-16T12:57:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1145224192 bytes 2024-09-16T12:57:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1194015744 bytes 2024-09-16T12:57:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1268705280 bytes 2024-09-16T12:57:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1334380544 bytes 2024-09-16T12:57:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1445782528 bytes 2024-09-16T12:57:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1468142592 bytes 2024-09-16T12:57:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1528127488 bytes 2024-09-16T12:57:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1573008384 bytes 2024-09-16T12:57:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1606133760 bytes 2024-09-16T12:57:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1637385216 bytes 2024-09-16T12:57:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1649544192 bytes 2024-09-16T12:57:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1694289920 bytes 2024-09-16T12:57:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1696441344 bytes 2024-09-16T12:57:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1729690624 bytes 2024-09-16T12:57:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1820015616 bytes 2024-09-16T12:57:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1858514944 bytes 2024-09-16T12:57:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1932006400 bytes 2024-09-16T12:57:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1932623872 bytes 2024-09-16T12:57:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1955159040 bytes 2024-09-16T12:57:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1974769664 bytes 2024-09-16T12:57:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1994452992 bytes 2024-09-16T12:57:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1998797824 bytes 2024-09-16T12:57:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2016586752 bytes 2024-09-16T12:57:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2058472448 bytes 2024-09-16T12:57:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2097046528 bytes 2024-09-16T12:57:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2150054912 bytes 2024-09-16T12:57:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2187201536 bytes 2024-09-16T12:57:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2217546752 bytes 2024-09-16T12:57:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2258491392 bytes 2024-09-16T12:57:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2274729984 bytes 2024-09-16T12:57:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2337805312 bytes 2024-09-16T12:57:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2393385984 bytes 2024-09-16T12:57:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2417737728 bytes 2024-09-16T12:57:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2450235392 bytes 2024-09-16T12:57:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2516184064 bytes 2024-09-16T12:57:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2540160000 bytes 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:57:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-16T12:57:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:57:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-09-16T12:57:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-16T12:57:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:57:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_19': 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 136c12ad2ab3 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.136c12ad2ab3 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:57:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:40 2024 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:40 2024 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:40 2024 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7c010caf-1b79-4274-b3dd-76aca299a0a4 2024-09-16T12:57:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:57:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:57:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-16T12:57:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T12:57:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:57:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T12:57:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:57:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:57:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 35182592 bytes 2024-09-16T12:57:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 69422080 bytes 2024-09-16T12:57:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 70040576 bytes 2024-09-16T12:57:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 110020608 bytes 2024-09-16T12:57:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 211610624 bytes 2024-09-16T12:57:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 311045120 bytes 2024-09-16T12:57:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 362321920 bytes 2024-09-16T12:57:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 367625216 bytes 2024-09-16T12:57:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 368268288 bytes 2024-09-16T12:57:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 372952064 bytes 2024-09-16T12:57:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 383752192 bytes 2024-09-16T12:57:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 445430784 bytes 2024-09-16T12:57:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 501406720 bytes 2024-09-16T12:57:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 567197696 bytes 2024-09-16T12:57:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 625905664 bytes 2024-09-16T12:57:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 665348096 bytes 2024-09-16T12:57:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 745998336 bytes 2024-09-16T12:57:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 811558912 bytes 2024-09-16T12:57:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 831491072 bytes 2024-09-16T12:57:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 882031616 bytes 2024-09-16T12:57:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 983984128 bytes 2024-09-16T12:57:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1022632960 bytes 2024-09-16T12:57:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1056066560 bytes 2024-09-16T12:57:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1142443008 bytes 2024-09-16T12:57:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1167901696 bytes 2024-09-16T12:57:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1184748544 bytes 2024-09-16T12:57:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1195532288 bytes 2024-09-16T12:57:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1241456640 bytes 2024-09-16T12:57:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1281432576 bytes 2024-09-16T12:57:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1299769344 bytes 2024-09-16T12:57:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1374580736 bytes 2024-09-16T12:57:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1450950656 bytes 2024-09-16T12:57:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1463331840 bytes 2024-09-16T12:57:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1479066624 bytes 2024-09-16T12:57:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1521334272 bytes 2024-09-16T12:57:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1573452800 bytes 2024-09-16T12:57:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1695036416 bytes 2024-09-16T12:57:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1769019392 bytes 2024-09-16T12:57:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1830957056 bytes 2024-09-16T12:57:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1862561792 bytes 2024-09-16T12:57:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1916992512 bytes 2024-09-16T12:57:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1946267648 bytes 2024-09-16T12:57:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1971687424 bytes 2024-09-16T12:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2054978560 bytes 2024-09-16T12:57:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2078518272 bytes 2024-09-16T12:57:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2079136768 bytes 2024-09-16T12:57:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2113224704 bytes 2024-09-16T12:57:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2175401984 bytes 2024-09-16T12:57:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2207600640 bytes 2024-09-16T12:57:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2249945088 bytes 2024-09-16T12:57:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2322285568 bytes 2024-09-16T12:57:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2352701440 bytes 2024-09-16T12:57:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2402353152 bytes 2024-09-16T12:57:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2422349824 bytes 2024-09-16T12:57:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2497790976 bytes 2024-09-16T12:57:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2580219904 bytes 2024-09-16T12:57:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2658843648 bytes 2024-09-16T12:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2692092928 bytes 2024-09-16T12:57:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2723543040 bytes 2024-09-16T12:57:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2761092096 bytes 2024-09-16T12:57:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2816872448 bytes 2024-09-16T12:57:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2817580032 bytes 2024-09-16T12:57:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2846200832 bytes 2024-09-16T12:57:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2897570816 bytes 2024-09-16T12:57:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3000341504 bytes 2024-09-16T12:57:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3014291456 bytes 2024-09-16T12:57:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3057266688 bytes 2024-09-16T12:57:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3111822336 bytes 2024-09-16T12:57:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3164485632 bytes 2024-09-16T12:57:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3200678912 bytes 2024-09-16T12:57:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3207691264 bytes 2024-09-16T12:57:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3277944832 bytes 2024-09-16T12:57:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3359325184 bytes 2024-09-16T12:57:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3412300800 bytes 2024-09-16T12:57:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3433333760 bytes 2024-09-16T12:57:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3460809728 bytes 2024-09-16T12:57:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3488215040 bytes 2024-09-16T12:57:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3506752512 bytes 2024-09-16T12:57:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3599396864 bytes 2024-09-16T12:57:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:57:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:57:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:57:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-16T12:57:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:57:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-09-16T12:57:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:57:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-16T12:57:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T12:57:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:57:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-16T12:57:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:57:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_20': 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1378840336ba 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1378840336ba 2024-09-16T12:57:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:55 2024 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:55 2024 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:55 2024 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 897043d8-7460-4263-8c01-2902509580ba 2024-09-16T12:57:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:57:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:57:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-16T12:57:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:57:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:57:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T12:57:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:57:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-16T12:57:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:57:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:57:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 595968 bytes 2024-09-16T12:57:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 40528896 bytes 2024-09-16T12:57:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 90276864 bytes 2024-09-16T12:57:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 180627456 bytes 2024-09-16T12:57:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 181214208 bytes 2024-09-16T12:57:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 215011328 bytes 2024-09-16T12:57:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 265837568 bytes 2024-09-16T12:57:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 321327104 bytes 2024-09-16T12:57:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 409434112 bytes 2024-09-16T12:57:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 458733568 bytes 2024-09-16T12:57:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 503936000 bytes 2024-09-16T12:57:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 536630272 bytes 2024-09-16T12:57:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 629437440 bytes 2024-09-16T12:57:58.848 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.456564418412128) > weight (0.25) 2024-09-16T12:57:58.848 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7129014295233026) > weight (0.25) 2024-09-16T12:57:58.848 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T12:57:58.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T12:57:58.849 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T12:57:58.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T12:57:58.847+0000 7f6b969f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 44725) UID: 1000 2024-09-16T12:57:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 670525440 bytes 2024-09-16T12:57:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 720556032 bytes 2024-09-16T12:57:59.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:57:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 747915264 bytes 2024-09-16T12:57:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 790628352 bytes 2024-09-16T12:57:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 866075648 bytes 2024-09-16T12:57:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 903928832 bytes 2024-09-16T12:57:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 904529920 bytes 2024-09-16T12:57:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 906631168 bytes 2024-09-16T12:58:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 934662144 bytes 2024-09-16T12:58:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1013155840 bytes 2024-09-16T12:58:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1034342400 bytes 2024-09-16T12:58:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1102045184 bytes 2024-09-16T12:58:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1122214912 bytes 2024-09-16T12:58:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1171772416 bytes 2024-09-16T12:58:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1205059584 bytes 2024-09-16T12:58:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1227469824 bytes 2024-09-16T12:58:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1248269312 bytes 2024-09-16T12:58:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1248893952 bytes 2024-09-16T12:58:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1268203520 bytes 2024-09-16T12:58:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1339379712 bytes 2024-09-16T12:58:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1414413312 bytes 2024-09-16T12:58:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1437835264 bytes 2024-09-16T12:58:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1477894144 bytes 2024-09-16T12:58:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1503132672 bytes 2024-09-16T12:58:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1531783168 bytes 2024-09-16T12:58:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1590909952 bytes 2024-09-16T12:58:02.850 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T12:58:02.850 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:58:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:58:02.851 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T12:58:02.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T12:58:02.851 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T12:58:02.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T12:58:02.854 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-16T12:58:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1652453376 bytes 2024-09-16T12:58:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1709461504 bytes 2024-09-16T12:58:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1776500736 bytes 2024-09-16T12:58:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1798735872 bytes 2024-09-16T12:58:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1846195200 bytes 2024-09-16T12:58:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1898572800 bytes 2024-09-16T12:58:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1923758080 bytes 2024-09-16T12:58:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1957298176 bytes 2024-09-16T12:58:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1983312896 bytes 2024-09-16T12:58:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2005284864 bytes 2024-09-16T12:58:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2082260992 bytes 2024-09-16T12:58:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2120296448 bytes 2024-09-16T12:58:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2150508544 bytes 2024-09-16T12:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2219609088 bytes 2024-09-16T12:58:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2289170432 bytes 2024-09-16T12:58:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2321327104 bytes 2024-09-16T12:58:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2322811904 bytes 2024-09-16T12:58:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2399908864 bytes 2024-09-16T12:58:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2463416320 bytes 2024-09-16T12:58:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2537507840 bytes 2024-09-16T12:58:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2603860992 bytes 2024-09-16T12:58:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2604474368 bytes 2024-09-16T12:58:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2628900864 bytes 2024-09-16T12:58:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2687291392 bytes 2024-09-16T12:58:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2697178112 bytes 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:58:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:58:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:58:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-16T12:58:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:58:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_21': 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13895a54ab7c 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13895a54ab7c 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:07 2024 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:07 2024 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:07 2024 2024-09-16T12:58:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:58:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:58:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5ec8e0c7-23d8-4f56-a4da-5c7e8d8a4880 2024-09-16T12:58:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T12:58:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:58:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T12:58:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:58:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:58:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 53746688 bytes 2024-09-16T12:58:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75261952 bytes 2024-09-16T12:58:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 136258560 bytes 2024-09-16T12:58:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 136893440 bytes 2024-09-16T12:58:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 188631040 bytes 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 227765248 bytes 2024-09-16T12:58:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 269063168 bytes 2024-09-16T12:58:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 269686784 bytes 2024-09-16T12:58:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 312022016 bytes 2024-09-16T12:58:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 390787072 bytes 2024-09-16T12:58:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 434370560 bytes 2024-09-16T12:58:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 484749312 bytes 2024-09-16T12:58:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 567188480 bytes 2024-09-16T12:58:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 651503616 bytes 2024-09-16T12:58:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 737190912 bytes 2024-09-16T12:58:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 737795072 bytes 2024-09-16T12:58:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 785997824 bytes 2024-09-16T12:58:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 814978048 bytes 2024-09-16T12:58:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 868519936 bytes 2024-09-16T12:58:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 907630592 bytes 2024-09-16T12:58:11.776 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6583639823896049) > weight (0.25) 2024-09-16T12:58:11.776 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T12:58:11.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T12:58:11.776 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T12:58:11.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T12:58:11.775+0000 7f45f5bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38179) UID: 1000 2024-09-16T12:58:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 918964224 bytes 2024-09-16T12:58:11.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T12:58:11.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 963067904 bytes 2024-09-16T12:58:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1017939968 bytes 2024-09-16T12:58:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1026135040 bytes 2024-09-16T12:58:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1075243008 bytes 2024-09-16T12:58:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1086999552 bytes 2024-09-16T12:58:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1148987392 bytes 2024-09-16T12:58:12.900 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:58:12.898+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dc264340 2024-09-16T12:58:12.901 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:58:12.899+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dc311040 2024-09-16T12:58:12.901 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T12:58:12.900+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dd3d45b0 2024-09-16T12:58:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1172899840 bytes 2024-09-16T12:58:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1228289024 bytes 2024-09-16T12:58:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1235000320 bytes 2024-09-16T12:58:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1246364672 bytes 2024-09-16T12:58:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1271347200 bytes 2024-09-16T12:58:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1322278912 bytes 2024-09-16T12:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1378315264 bytes 2024-09-16T12:58:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1421584384 bytes 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-16T12:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-09-16T12:58:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:58:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-16T12:58:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T12:58:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-16T12:58:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:58:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-16T12:58:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_22': 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1394c2a4ab2e 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1394c2a4ab2e 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:14 2024 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:14 2024 2024-09-16T12:58:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:14 2024 2024-09-16T12:58:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:58:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:58:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 4135e1aa-5a4e-4f22-a4a4-d979c6386191 2024-09-16T12:58:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:58:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:58:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-16T12:58:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T12:58:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:58:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T12:58:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:58:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:58:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 15416320 bytes 2024-09-16T12:58:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 38230016 bytes 2024-09-16T12:58:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 171891712 bytes 2024-09-16T12:58:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 248001536 bytes 2024-09-16T12:58:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 254949376 bytes 2024-09-16T12:58:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 268347392 bytes 2024-09-16T12:58:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 320637952 bytes 2024-09-16T12:58:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 373067776 bytes 2024-09-16T12:58:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 373697536 bytes 2024-09-16T12:58:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 491986944 bytes 2024-09-16T12:58:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 517548032 bytes 2024-09-16T12:58:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 531019776 bytes 2024-09-16T12:58:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 541182976 bytes 2024-09-16T12:58:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 556930048 bytes 2024-09-16T12:58:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 574110720 bytes 2024-09-16T12:58:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 587491328 bytes 2024-09-16T12:58:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 615707648 bytes 2024-09-16T12:58:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 637977600 bytes 2024-09-16T12:58:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 695236608 bytes 2024-09-16T12:58:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 704304128 bytes 2024-09-16T12:58:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 746909696 bytes 2024-09-16T12:58:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 785859584 bytes 2024-09-16T12:58:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 860262400 bytes 2024-09-16T12:58:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 902535168 bytes 2024-09-16T12:58:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 925558784 bytes 2024-09-16T12:58:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1003792384 bytes 2024-09-16T12:58:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1052355584 bytes 2024-09-16T12:58:19.780 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T12:58:19.780 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T12:58:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:58:19.781 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T12:58:19.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T12:58:19.781 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T12:58:19.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T12:58:19.784 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-16T12:58:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1088509952 bytes 2024-09-16T12:58:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1144685568 bytes 2024-09-16T12:58:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1148487680 bytes 2024-09-16T12:58:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1158678528 bytes 2024-09-16T12:58:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1201744896 bytes 2024-09-16T12:58:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1240811520 bytes 2024-09-16T12:58:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1307189248 bytes 2024-09-16T12:58:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1307803648 bytes 2024-09-16T12:58:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1369121792 bytes 2024-09-16T12:58:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1401269248 bytes 2024-09-16T12:58:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1430903808 bytes 2024-09-16T12:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1514935296 bytes 2024-09-16T12:58:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1564798976 bytes 2024-09-16T12:58:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1658734592 bytes 2024-09-16T12:58:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1743553536 bytes 2024-09-16T12:58:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1760472064 bytes 2024-09-16T12:58:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1787870208 bytes 2024-09-16T12:58:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1845821440 bytes 2024-09-16T12:58:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1892473856 bytes 2024-09-16T12:58:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1958290432 bytes 2024-09-16T12:58:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1975514112 bytes 2024-09-16T12:58:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2031564800 bytes 2024-09-16T12:58:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2074865664 bytes 2024-09-16T12:58:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2153519104 bytes 2024-09-16T12:58:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2183823360 bytes 2024-09-16T12:58:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2246681600 bytes 2024-09-16T12:58:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2282634240 bytes 2024-09-16T12:58:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2287163392 bytes 2024-09-16T12:58:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2323458048 bytes 2024-09-16T12:58:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2352418816 bytes 2024-09-16T12:58:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2403875840 bytes 2024-09-16T12:58:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2458288128 bytes 2024-09-16T12:58:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2502161408 bytes 2024-09-16T12:58:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2537263104 bytes 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2561491968 bytes 2024-09-16T12:58:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2590391296 bytes 2024-09-16T12:58:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2634833920 bytes 2024-09-16T12:58:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2671124480 bytes 2024-09-16T12:58:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2700249088 bytes 2024-09-16T12:58:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2722013184 bytes 2024-09-16T12:58:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2722647040 bytes 2024-09-16T12:58:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2786838528 bytes 2024-09-16T12:58:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2836155392 bytes 2024-09-16T12:58:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2863425536 bytes 2024-09-16T12:58:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2900421632 bytes 2024-09-16T12:58:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2953371648 bytes 2024-09-16T12:58:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3045054464 bytes 2024-09-16T12:58:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3067509760 bytes 2024-09-16T12:58:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3119547392 bytes 2024-09-16T12:58:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3129690112 bytes 2024-09-16T12:58:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3151866880 bytes 2024-09-16T12:58:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3174930432 bytes 2024-09-16T12:58:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3239227392 bytes 2024-09-16T12:58:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3268020224 bytes 2024-09-16T12:58:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3293339648 bytes 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-16T12:58:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-09-16T12:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:58:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-16T12:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T12:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-16T12:58:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:58:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_23': 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13a7a3c5296c 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13a7a3c5296c 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:58:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:29 2024 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:29 2024 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:29 2024 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: deee2aeb-a92c-4e22-ab10-fb00148a0183 2024-09-16T12:58:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T12:58:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:58:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T12:58:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:58:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:58:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 29100032 bytes 2024-09-16T12:58:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75385856 bytes 2024-09-16T12:58:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 114052096 bytes 2024-09-16T12:58:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 168681472 bytes 2024-09-16T12:58:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 222188544 bytes 2024-09-16T12:58:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 265847808 bytes 2024-09-16T12:58:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 293518336 bytes 2024-09-16T12:58:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 388221952 bytes 2024-09-16T12:58:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 421374976 bytes 2024-09-16T12:58:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 424419328 bytes 2024-09-16T12:58:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 446493696 bytes 2024-09-16T12:58:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 489286656 bytes 2024-09-16T12:58:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 540658688 bytes 2024-09-16T12:58:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 571125760 bytes 2024-09-16T12:58:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 586550272 bytes 2024-09-16T12:58:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 638493696 bytes 2024-09-16T12:58:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 694537216 bytes 2024-09-16T12:58:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 776204288 bytes 2024-09-16T12:58:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 826757120 bytes 2024-09-16T12:58:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 852560896 bytes 2024-09-16T12:58:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 935737344 bytes 2024-09-16T12:58:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 960598016 bytes 2024-09-16T12:58:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 995658752 bytes 2024-09-16T12:58:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1027425280 bytes 2024-09-16T12:58:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1072555008 bytes 2024-09-16T12:58:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1108762624 bytes 2024-09-16T12:58:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1109393408 bytes 2024-09-16T12:58:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1116647424 bytes 2024-09-16T12:58:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1155935232 bytes 2024-09-16T12:58:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1176988672 bytes 2024-09-16T12:58:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1217166336 bytes 2024-09-16T12:58:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1238061056 bytes 2024-09-16T12:58:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1272546304 bytes 2024-09-16T12:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1311400960 bytes 2024-09-16T12:58:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1363495936 bytes 2024-09-16T12:58:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1372813312 bytes 2024-09-16T12:58:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1443459072 bytes 2024-09-16T12:58:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1483427840 bytes 2024-09-16T12:58:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1524656128 bytes 2024-09-16T12:58:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1592124416 bytes 2024-09-16T12:58:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1616205824 bytes 2024-09-16T12:58:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1626736640 bytes 2024-09-16T12:58:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1651366912 bytes 2024-09-16T12:58:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1683271680 bytes 2024-09-16T12:58:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1734207488 bytes 2024-09-16T12:58:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1767829504 bytes 2024-09-16T12:58:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1805851648 bytes 2024-09-16T12:58:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1832993792 bytes 2024-09-16T12:58:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1875079168 bytes 2024-09-16T12:58:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1916862464 bytes 2024-09-16T12:58:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1929108480 bytes 2024-09-16T12:58:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1978021888 bytes 2024-09-16T12:58:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2009265152 bytes 2024-09-16T12:58:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2022998016 bytes 2024-09-16T12:58:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2067001344 bytes 2024-09-16T12:58:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2099618816 bytes 2024-09-16T12:58:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2124944384 bytes 2024-09-16T12:58:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2152030208 bytes 2024-09-16T12:58:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2198425600 bytes 2024-09-16T12:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2280828928 bytes 2024-09-16T12:58:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2353097728 bytes 2024-09-16T12:58:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2369393664 bytes 2024-09-16T12:58:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2431631360 bytes 2024-09-16T12:58:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2452386816 bytes 2024-09-16T12:58:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2482107392 bytes 2024-09-16T12:58:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2525061120 bytes 2024-09-16T12:58:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2525658112 bytes 2024-09-16T12:58:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2538717184 bytes 2024-09-16T12:58:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2560140288 bytes 2024-09-16T12:58:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2607407104 bytes 2024-09-16T12:58:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2620791808 bytes 2024-09-16T12:58:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2658314240 bytes 2024-09-16T12:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2682854400 bytes 2024-09-16T12:58:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2782052352 bytes 2024-09-16T12:58:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2812143616 bytes 2024-09-16T12:58:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2833054720 bytes 2024-09-16T12:58:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 2885698560 bytes 2024-09-16T12:58:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 2914180096 bytes 2024-09-16T12:58:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:58:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-09-16T12:58:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:58:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-16T12:58:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T12:58:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-16T12:58:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:58:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-16T12:58:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_24': 2024-09-16T12:58:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13b356cf29be 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13b356cf29be 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:43 2024 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:43 2024 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:43 2024 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:58:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:58:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d84f0610-c1af-4671-8b4a-aeb184619555 2024-09-16T12:58:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T12:58:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:58:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T12:58:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:58:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:58:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 33230848 bytes 2024-09-16T12:58:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 112058368 bytes 2024-09-16T12:58:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 131106816 bytes 2024-09-16T12:58:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 131732480 bytes 2024-09-16T12:58:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 173062144 bytes 2024-09-16T12:58:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 177611776 bytes 2024-09-16T12:58:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 194358272 bytes 2024-09-16T12:58:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 247198720 bytes 2024-09-16T12:58:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 286343168 bytes 2024-09-16T12:58:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 314220544 bytes 2024-09-16T12:58:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 394206208 bytes 2024-09-16T12:58:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 481831936 bytes 2024-09-16T12:58:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 556003328 bytes 2024-09-16T12:58:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 599411712 bytes 2024-09-16T12:58:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 600020992 bytes 2024-09-16T12:58:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 660559872 bytes 2024-09-16T12:58:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 702161920 bytes 2024-09-16T12:58:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 798696448 bytes 2024-09-16T12:58:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 878105600 bytes 2024-09-16T12:58:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 904887296 bytes 2024-09-16T12:58:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 938508288 bytes 2024-09-16T12:58:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 982938624 bytes 2024-09-16T12:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1017360384 bytes 2024-09-16T12:58:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1077771264 bytes 2024-09-16T12:58:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1085696000 bytes 2024-09-16T12:58:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1122571264 bytes 2024-09-16T12:58:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1151767552 bytes 2024-09-16T12:58:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1235220480 bytes 2024-09-16T12:58:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1281265664 bytes 2024-09-16T12:58:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1294366720 bytes 2024-09-16T12:58:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1324644352 bytes 2024-09-16T12:58:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1393642496 bytes 2024-09-16T12:58:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1409213440 bytes 2024-09-16T12:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1426845696 bytes 2024-09-16T12:58:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1502891008 bytes 2024-09-16T12:58:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1503472640 bytes 2024-09-16T12:58:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1540173824 bytes 2024-09-16T12:58:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1603078144 bytes 2024-09-16T12:58:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1614734336 bytes 2024-09-16T12:58:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1619386368 bytes 2024-09-16T12:58:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1632324608 bytes 2024-09-16T12:58:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1646102528 bytes 2024-09-16T12:58:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1701809152 bytes 2024-09-16T12:58:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1739151360 bytes 2024-09-16T12:58:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1757962240 bytes 2024-09-16T12:58:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1771979776 bytes 2024-09-16T12:58:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1812295680 bytes 2024-09-16T12:58:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1837743104 bytes 2024-09-16T12:58:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1838360576 bytes 2024-09-16T12:58:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1902450688 bytes 2024-09-16T12:58:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1952264192 bytes 2024-09-16T12:58:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1958652928 bytes 2024-09-16T12:58:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1991468032 bytes 2024-09-16T12:58:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2044759040 bytes 2024-09-16T12:58:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2118466560 bytes 2024-09-16T12:58:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2119095296 bytes 2024-09-16T12:58:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2168329216 bytes 2024-09-16T12:58:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2195345408 bytes 2024-09-16T12:58:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2254360576 bytes 2024-09-16T12:58:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2305683456 bytes 2024-09-16T12:58:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2307148800 bytes 2024-09-16T12:58:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2372654080 bytes 2024-09-16T12:58:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2414825472 bytes 2024-09-16T12:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2434445312 bytes 2024-09-16T12:58:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2486749184 bytes 2024-09-16T12:58:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2487342080 bytes 2024-09-16T12:58:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2529075200 bytes 2024-09-16T12:58:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2596820992 bytes 2024-09-16T12:58:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2634053632 bytes 2024-09-16T12:58:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2679889920 bytes 2024-09-16T12:58:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2732015616 bytes 2024-09-16T12:58:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2746546176 bytes 2024-09-16T12:58:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2778366976 bytes 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T12:58:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:58:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:58:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:58:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:58:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T12:58:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:58:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-16T12:58:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:58:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_25': 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13bf783951d6 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13bf783951d6 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:56 2024 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:56 2024 2024-09-16T12:58:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:56 2024 2024-09-16T12:58:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:58:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:58:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: dde89dcd-ca3e-4c90-a12d-78663eac03bf 2024-09-16T12:58:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T12:58:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:58:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T12:58:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:58:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:58:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 37338112 bytes 2024-09-16T12:58:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 93365248 bytes 2024-09-16T12:58:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 113384448 bytes 2024-09-16T12:58:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 178911232 bytes 2024-09-16T12:58:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 192535552 bytes 2024-09-16T12:58:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 222233600 bytes 2024-09-16T12:58:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 299018240 bytes 2024-09-16T12:58:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 370758656 bytes 2024-09-16T12:58:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 435263488 bytes 2024-09-16T12:58:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 451515392 bytes 2024-09-16T12:58:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 463556608 bytes 2024-09-16T12:58:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 536415232 bytes 2024-09-16T12:58:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 564442112 bytes 2024-09-16T12:58:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 572347392 bytes 2024-09-16T12:59:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 606020608 bytes 2024-09-16T12:59:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 639001600 bytes 2024-09-16T12:59:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 676334592 bytes 2024-09-16T12:59:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 753245184 bytes 2024-09-16T12:59:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 801104896 bytes 2024-09-16T12:59:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 851836928 bytes 2024-09-16T12:59:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 915957760 bytes 2024-09-16T12:59:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 916574208 bytes 2024-09-16T12:59:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 959877120 bytes 2024-09-16T12:59:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 971735040 bytes 2024-09-16T12:59:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1004591104 bytes 2024-09-16T12:59:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1046755328 bytes 2024-09-16T12:59:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1053802496 bytes 2024-09-16T12:59:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1110151168 bytes 2024-09-16T12:59:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1167212544 bytes 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-16T12:59:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_26': 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13ca95c155f9 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13ca95c155f9 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:02 2024 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:02 2024 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:02 2024 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3b3040ad-963e-4bf9-8b3f-18deb1f76778 2024-09-16T12:59:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T12:59:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:59:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 74987520 bytes 2024-09-16T12:59:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 131453952 bytes 2024-09-16T12:59:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 142060544 bytes 2024-09-16T12:59:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 164599808 bytes 2024-09-16T12:59:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 178956288 bytes 2024-09-16T12:59:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 239494144 bytes 2024-09-16T12:59:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 278871040 bytes 2024-09-16T12:59:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 391829504 bytes 2024-09-16T12:59:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 439988224 bytes 2024-09-16T12:59:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 562982912 bytes 2024-09-16T12:59:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 590455808 bytes 2024-09-16T12:59:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 646157312 bytes 2024-09-16T12:59:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 697283584 bytes 2024-09-16T12:59:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 739420160 bytes 2024-09-16T12:59:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 778316800 bytes 2024-09-16T12:59:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 797602816 bytes 2024-09-16T12:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 813380608 bytes 2024-09-16T12:59:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 829232128 bytes 2024-09-16T12:59:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 844982272 bytes 2024-09-16T12:59:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 894965760 bytes 2024-09-16T12:59:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1041659904 bytes 2024-09-16T12:59:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1093780480 bytes 2024-09-16T12:59:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1117346816 bytes 2024-09-16T12:59:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1151870976 bytes 2024-09-16T12:59:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1155116032 bytes 2024-09-16T12:59:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1171488768 bytes 2024-09-16T12:59:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1172792320 bytes 2024-09-16T12:59:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1194055680 bytes 2024-09-16T12:59:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1258424320 bytes 2024-09-16T12:59:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1285533696 bytes 2024-09-16T12:59:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1295879168 bytes 2024-09-16T12:59:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1319299072 bytes 2024-09-16T12:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1342027776 bytes 2024-09-16T12:59:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1383695360 bytes 2024-09-16T12:59:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1400934400 bytes 2024-09-16T12:59:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1404220416 bytes 2024-09-16T12:59:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1444084736 bytes 2024-09-16T12:59:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1491955712 bytes 2024-09-16T12:59:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1581120512 bytes 2024-09-16T12:59:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1596457984 bytes 2024-09-16T12:59:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-16T12:59:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-16T12:59:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_27': 2024-09-16T12:59:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13d6bbb77c50 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13d6bbb77c50 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:10 2024 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:10 2024 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:10 2024 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6e695d31-7fbb-4a6d-bf1c-da34f746931f 2024-09-16T12:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T12:59:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-16T12:59:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 37906432 bytes 2024-09-16T12:59:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 97713152 bytes 2024-09-16T12:59:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 115482624 bytes 2024-09-16T12:59:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 173194240 bytes 2024-09-16T12:59:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 281761792 bytes 2024-09-16T12:59:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 294030336 bytes 2024-09-16T12:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 371517440 bytes 2024-09-16T12:59:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 372131840 bytes 2024-09-16T12:59:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 450721792 bytes 2024-09-16T12:59:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 479336448 bytes 2024-09-16T12:59:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 514513920 bytes 2024-09-16T12:59:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 550677504 bytes 2024-09-16T12:59:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 563892224 bytes 2024-09-16T12:59:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 613534720 bytes 2024-09-16T12:59:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 698635264 bytes 2024-09-16T12:59:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 742392832 bytes 2024-09-16T12:59:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 801328128 bytes 2024-09-16T12:59:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 853173248 bytes 2024-09-16T12:59:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 872403968 bytes 2024-09-16T12:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 917555200 bytes 2024-09-16T12:59:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 960918528 bytes 2024-09-16T12:59:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1003329536 bytes 2024-09-16T12:59:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1039325184 bytes 2024-09-16T12:59:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1092308992 bytes 2024-09-16T12:59:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1110576128 bytes 2024-09-16T12:59:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1136478208 bytes 2024-09-16T12:59:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1179643904 bytes 2024-09-16T12:59:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1202560000 bytes 2024-09-16T12:59:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1259950080 bytes 2024-09-16T12:59:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1280936960 bytes 2024-09-16T12:59:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1338449920 bytes 2024-09-16T12:59:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1382989824 bytes 2024-09-16T12:59:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1417742336 bytes 2024-09-16T12:59:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1463435264 bytes 2024-09-16T12:59:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1502035968 bytes 2024-09-16T12:59:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1545729024 bytes 2024-09-16T12:59:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1596054528 bytes 2024-09-16T12:59:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1644210176 bytes 2024-09-16T12:59:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1698251776 bytes 2024-09-16T12:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1711837184 bytes 2024-09-16T12:59:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1717652480 bytes 2024-09-16T12:59:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1744000000 bytes 2024-09-16T12:59:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1795745792 bytes 2024-09-16T12:59:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1898969088 bytes 2024-09-16T12:59:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T12:59:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-16T12:59:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-16T12:59:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_28': 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13e22d572850 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13e22d572850 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:19 2024 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:19 2024 2024-09-16T12:59:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:19 2024 2024-09-16T12:59:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ed51faf8-e2c3-44a4-9d09-787be9fd78f9 2024-09-16T12:59:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T12:59:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T12:59:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 31348736 bytes 2024-09-16T12:59:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 37621760 bytes 2024-09-16T12:59:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 118626304 bytes 2024-09-16T12:59:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 144763904 bytes 2024-09-16T12:59:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 226136064 bytes 2024-09-16T12:59:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 323481600 bytes 2024-09-16T12:59:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 397320192 bytes 2024-09-16T12:59:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 453606400 bytes 2024-09-16T12:59:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 481225728 bytes 2024-09-16T12:59:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 534535168 bytes 2024-09-16T12:59:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 561747968 bytes 2024-09-16T12:59:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 647488512 bytes 2024-09-16T12:59:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 720336896 bytes 2024-09-16T12:59:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 799201280 bytes 2024-09-16T12:59:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 810758144 bytes 2024-09-16T12:59:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 854647808 bytes 2024-09-16T12:59:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 855256064 bytes 2024-09-16T12:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 866335744 bytes 2024-09-16T12:59:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 950455296 bytes 2024-09-16T12:59:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 990838784 bytes 2024-09-16T12:59:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1029918720 bytes 2024-09-16T12:59:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1096630272 bytes 2024-09-16T12:59:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1173046272 bytes 2024-09-16T12:59:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1244277760 bytes 2024-09-16T12:59:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1291622400 bytes 2024-09-16T12:59:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1374154752 bytes 2024-09-16T12:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1399628800 bytes 2024-09-16T12:59:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1411939328 bytes 2024-09-16T12:59:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1436018688 bytes 2024-09-16T12:59:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1482714112 bytes 2024-09-16T12:59:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1563819008 bytes 2024-09-16T12:59:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1590164480 bytes 2024-09-16T12:59:26.855 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.49212263465791806) > weight (0.25) 2024-09-16T12:59:26.855 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9671883167072255) > weight (0.25) 2024-09-16T12:59:26.855 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7596569295525814) > weight (0.25) 2024-09-16T12:59:26.855 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7237525546897431) > weight (0.25) 2024-09-16T12:59:26.855 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-16T12:59:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1622842368 bytes 2024-09-16T12:59:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1659328512 bytes 2024-09-16T12:59:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1733365760 bytes 2024-09-16T12:59:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1808791552 bytes 2024-09-16T12:59:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1878765568 bytes 2024-09-16T12:59:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1913545728 bytes 2024-09-16T12:59:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1954499584 bytes 2024-09-16T12:59:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1978608640 bytes 2024-09-16T12:59:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2008374272 bytes 2024-09-16T12:59:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2037031936 bytes 2024-09-16T12:59:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2046534656 bytes 2024-09-16T12:59:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2093612032 bytes 2024-09-16T12:59:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2104129536 bytes 2024-09-16T12:59:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2134765568 bytes 2024-09-16T12:59:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2191705088 bytes 2024-09-16T12:59:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2219457536 bytes 2024-09-16T12:59:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2239294464 bytes 2024-09-16T12:59:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2301863936 bytes 2024-09-16T12:59:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2330635264 bytes 2024-09-16T12:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2360775680 bytes 2024-09-16T12:59:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2398014464 bytes 2024-09-16T12:59:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2414236672 bytes 2024-09-16T12:59:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2453264384 bytes 2024-09-16T12:59:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2466658304 bytes 2024-09-16T12:59:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2515845120 bytes 2024-09-16T12:59:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2588617728 bytes 2024-09-16T12:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2642101248 bytes 2024-09-16T12:59:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2662237184 bytes 2024-09-16T12:59:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2710165504 bytes 2024-09-16T12:59:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2754306048 bytes 2024-09-16T12:59:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2763296768 bytes 2024-09-16T12:59:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2793110528 bytes 2024-09-16T12:59:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2800827392 bytes 2024-09-16T12:59:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2818121728 bytes 2024-09-16T12:59:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2827625472 bytes 2024-09-16T12:59:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2852576256 bytes 2024-09-16T12:59:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2891213824 bytes 2024-09-16T12:59:32.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2911987712 bytes 2024-09-16T12:59:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-09-16T12:59:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-16T12:59:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T12:59:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-16T12:59:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-16T12:59:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_29': 2024-09-16T12:59:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13ee6ab2201c 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13ee6ab2201c 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:32 2024 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:32 2024 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:32 2024 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 21593e92-6a4d-4963-8412-532dd99efda2 2024-09-16T12:59:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T12:59:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T12:59:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 72821760 bytes 2024-09-16T12:59:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 112588800 bytes 2024-09-16T12:59:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 171260928 bytes 2024-09-16T12:59:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 240678912 bytes 2024-09-16T12:59:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 274899968 bytes 2024-09-16T12:59:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 308459520 bytes 2024-09-16T12:59:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 346519552 bytes 2024-09-16T12:59:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 377846784 bytes 2024-09-16T12:59:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 467763200 bytes 2024-09-16T12:59:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 488701952 bytes 2024-09-16T12:59:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 539667456 bytes 2024-09-16T12:59:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 608901120 bytes 2024-09-16T12:59:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 657392640 bytes 2024-09-16T12:59:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 713499648 bytes 2024-09-16T12:59:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 728432640 bytes 2024-09-16T12:59:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 741818368 bytes 2024-09-16T12:59:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 771099648 bytes 2024-09-16T12:59:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 854310912 bytes 2024-09-16T12:59:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 866992128 bytes 2024-09-16T12:59:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 880096256 bytes 2024-09-16T12:59:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 897998848 bytes 2024-09-16T12:59:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1014152192 bytes 2024-09-16T12:59:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1059123200 bytes 2024-09-16T12:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1099206656 bytes 2024-09-16T12:59:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1160484864 bytes 2024-09-16T12:59:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1161629696 bytes 2024-09-16T12:59:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1177867264 bytes 2024-09-16T12:59:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1250762752 bytes 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-16T12:59:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-16T12:59:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_30': 2024-09-16T12:59:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13fae85ece37 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13fae85ece37 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:38 2024 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:38 2024 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:38 2024 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 795f020a-5b0a-43d5-a652-615b46e9624a 2024-09-16T12:59:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T12:59:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T12:59:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 45734912 bytes 2024-09-16T12:59:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 48583680 bytes 2024-09-16T12:59:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 70640640 bytes 2024-09-16T12:59:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 126553088 bytes 2024-09-16T12:59:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 190316544 bytes 2024-09-16T12:59:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 193783808 bytes 2024-09-16T12:59:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 240224256 bytes 2024-09-16T12:59:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 305668096 bytes 2024-09-16T12:59:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 363177984 bytes 2024-09-16T12:59:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 385563648 bytes 2024-09-16T12:59:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 411755520 bytes 2024-09-16T12:59:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 490805248 bytes 2024-09-16T12:59:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 581966848 bytes 2024-09-16T12:59:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 582583296 bytes 2024-09-16T12:59:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 608478208 bytes 2024-09-16T12:59:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 652448768 bytes 2024-09-16T12:59:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 714859520 bytes 2024-09-16T12:59:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 734233600 bytes 2024-09-16T12:59:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 734813184 bytes 2024-09-16T12:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 767961088 bytes 2024-09-16T12:59:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 822348800 bytes 2024-09-16T12:59:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 868077568 bytes 2024-09-16T12:59:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 916757504 bytes 2024-09-16T12:59:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 996333568 bytes 2024-09-16T12:59:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1060732928 bytes 2024-09-16T12:59:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1106975744 bytes 2024-09-16T12:59:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1163797504 bytes 2024-09-16T12:59:45.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1186834432 bytes 2024-09-16T12:59:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1251341312 bytes 2024-09-16T12:59:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1287194624 bytes 2024-09-16T12:59:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1308069888 bytes 2024-09-16T12:59:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1339710464 bytes 2024-09-16T12:59:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1399677952 bytes 2024-09-16T12:59:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1462960128 bytes 2024-09-16T12:59:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1500017664 bytes 2024-09-16T12:59:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1512745984 bytes 2024-09-16T12:59:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1520177152 bytes 2024-09-16T12:59:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1624731648 bytes 2024-09-16T12:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1665106944 bytes 2024-09-16T12:59:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1715841024 bytes 2024-09-16T12:59:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1745169408 bytes 2024-09-16T12:59:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1779575808 bytes 2024-09-16T12:59:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1795533824 bytes 2024-09-16T12:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1830118400 bytes 2024-09-16T12:59:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1912810496 bytes 2024-09-16T12:59:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1950629888 bytes 2024-09-16T12:59:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1982726144 bytes 2024-09-16T12:59:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1983330304 bytes 2024-09-16T12:59:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2037472256 bytes 2024-09-16T12:59:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2105454592 bytes 2024-09-16T12:59:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2157595648 bytes 2024-09-16T12:59:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2185749504 bytes 2024-09-16T12:59:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2205763584 bytes 2024-09-16T12:59:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2252926976 bytes 2024-09-16T12:59:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2271491072 bytes 2024-09-16T12:59:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2337519616 bytes 2024-09-16T12:59:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2374974464 bytes 2024-09-16T12:59:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2445617152 bytes 2024-09-16T12:59:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2466833408 bytes 2024-09-16T12:59:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2510580736 bytes 2024-09-16T12:59:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2550872064 bytes 2024-09-16T12:59:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2576074752 bytes 2024-09-16T12:59:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2582439936 bytes 2024-09-16T12:59:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2638528512 bytes 2024-09-16T12:59:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2678953984 bytes 2024-09-16T12:59:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2704955392 bytes 2024-09-16T12:59:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2717096960 bytes 2024-09-16T12:59:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2760865792 bytes 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T12:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-09-16T12:59:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T12:59:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-16T12:59:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T12:59:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T12:59:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-16T12:59:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T12:59:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_31': 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1406f2f90ceb 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1406f2f90ceb 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T12:59:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:51 2024 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:51 2024 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:51 2024 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: eeb2f380-5802-4457-936c-f678d743f57e 2024-09-16T12:59:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T12:59:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T12:59:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-16T12:59:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T12:59:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T12:59:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T12:59:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T12:59:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T12:59:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T12:59:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-16T12:59:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 19816448 bytes 2024-09-16T12:59:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 39266304 bytes 2024-09-16T12:59:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 84711424 bytes 2024-09-16T12:59:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 122935296 bytes 2024-09-16T12:59:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 195932160 bytes 2024-09-16T12:59:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 250555392 bytes 2024-09-16T12:59:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 281411584 bytes 2024-09-16T12:59:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 300129280 bytes 2024-09-16T12:59:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 389108736 bytes 2024-09-16T12:59:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 400574464 bytes 2024-09-16T12:59:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 420099072 bytes 2024-09-16T12:59:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 466383872 bytes 2024-09-16T12:59:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 502644736 bytes 2024-09-16T12:59:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 519568384 bytes 2024-09-16T12:59:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 566640640 bytes 2024-09-16T12:59:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 588057600 bytes 2024-09-16T12:59:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 605119488 bytes 2024-09-16T12:59:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 668300288 bytes 2024-09-16T12:59:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 693846016 bytes 2024-09-16T12:59:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 713057280 bytes 2024-09-16T12:59:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 719023104 bytes 2024-09-16T12:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 770423808 bytes 2024-09-16T12:59:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 814979072 bytes 2024-09-16T12:59:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 873682944 bytes 2024-09-16T12:59:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 906353664 bytes 2024-09-16T12:59:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 952201216 bytes 2024-09-16T12:59:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 996254720 bytes 2024-09-16T12:59:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1041633280 bytes 2024-09-16T12:59:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1070453760 bytes 2024-09-16T12:59:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1158777856 bytes 2024-09-16T12:59:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1193141248 bytes 2024-09-16T12:59:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1232696320 bytes 2024-09-16T12:59:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1264230400 bytes 2024-09-16T12:59:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1264822272 bytes 2024-09-16T12:59:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1279176704 bytes 2024-09-16T12:59:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1296272384 bytes 2024-09-16T12:59:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1346761728 bytes 2024-09-16T12:59:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1389309952 bytes 2024-09-16T12:59:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1441351680 bytes 2024-09-16T12:59:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1470052352 bytes 2024-09-16T12:59:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1493136384 bytes 2024-09-16T12:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1549432832 bytes 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:00:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-16T13:00:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_32': 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1412d9088a36 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1412d9088a36 2024-09-16T13:00:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:00 2024 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:00 2024 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:00 2024 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7bac516b-dc94-4b85-9cd0-8bcb96179e9a 2024-09-16T13:00:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:00:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-16T13:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 73111552 bytes 2024-09-16T13:00:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 130276352 bytes 2024-09-16T13:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 209911808 bytes 2024-09-16T13:00:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 282919936 bytes 2024-09-16T13:00:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 283543552 bytes 2024-09-16T13:00:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 306029568 bytes 2024-09-16T13:00:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 403449856 bytes 2024-09-16T13:00:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 480317440 bytes 2024-09-16T13:00:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 537956352 bytes 2024-09-16T13:00:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 607203328 bytes 2024-09-16T13:00:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 633826304 bytes 2024-09-16T13:00:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 677561344 bytes 2024-09-16T13:00:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 735410176 bytes 2024-09-16T13:00:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 760494080 bytes 2024-09-16T13:00:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 785116160 bytes 2024-09-16T13:00:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 807865344 bytes 2024-09-16T13:00:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 837803008 bytes 2024-09-16T13:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 872582144 bytes 2024-09-16T13:00:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 905677824 bytes 2024-09-16T13:00:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 907581440 bytes 2024-09-16T13:00:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 908224512 bytes 2024-09-16T13:00:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1031158784 bytes 2024-09-16T13:00:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1076485120 bytes 2024-09-16T13:00:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1189929984 bytes 2024-09-16T13:00:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1259296768 bytes 2024-09-16T13:00:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1279953920 bytes 2024-09-16T13:00:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1339816960 bytes 2024-09-16T13:00:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1392594944 bytes 2024-09-16T13:00:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1395929088 bytes 2024-09-16T13:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1463546880 bytes 2024-09-16T13:00:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1523503104 bytes 2024-09-16T13:00:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1569434624 bytes 2024-09-16T13:00:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1630640128 bytes 2024-09-16T13:00:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1654096896 bytes 2024-09-16T13:00:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1672545280 bytes 2024-09-16T13:00:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1685352448 bytes 2024-09-16T13:00:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1721927680 bytes 2024-09-16T13:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1755133952 bytes 2024-09-16T13:00:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1755752448 bytes 2024-09-16T13:00:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1815858176 bytes 2024-09-16T13:00:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1851428864 bytes 2024-09-16T13:00:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1952031744 bytes 2024-09-16T13:00:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1961124864 bytes 2024-09-16T13:00:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2012310528 bytes 2024-09-16T13:00:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2034849792 bytes 2024-09-16T13:00:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2118254592 bytes 2024-09-16T13:00:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2178942976 bytes 2024-09-16T13:00:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2179566592 bytes 2024-09-16T13:00:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2213067776 bytes 2024-09-16T13:00:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2261665792 bytes 2024-09-16T13:00:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2291303424 bytes 2024-09-16T13:00:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2332912640 bytes 2024-09-16T13:00:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2399896576 bytes 2024-09-16T13:00:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2419892224 bytes 2024-09-16T13:00:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2482718720 bytes 2024-09-16T13:00:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2509420544 bytes 2024-09-16T13:00:10.785 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5887153276147552) > weight (0.25) 2024-09-16T13:00:10.785 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8263880038886879) > weight (0.25) 2024-09-16T13:00:10.785 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T13:00:10.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T13:00:10.786 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T13:00:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:00:10.785+0000 7f13e91f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 48394) UID: 1000 2024-09-16T13:00:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2583386112 bytes 2024-09-16T13:00:10.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:00:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2625638400 bytes 2024-09-16T13:00:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2694476800 bytes 2024-09-16T13:00:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2695102464 bytes 2024-09-16T13:00:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2783974400 bytes 2024-09-16T13:00:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-09-16T13:00:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-16T13:00:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:00:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-16T13:00:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_33': 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 141e490b0245 2024-09-16T13:00:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.141e490b0245 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:11 2024 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:11 2024 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:11 2024 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 686e94e1-8f41-41d6-998f-9112a95df1d3 2024-09-16T13:00:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-16T13:00:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:00:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:00:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 43857920 bytes 2024-09-16T13:00:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 74909696 bytes 2024-09-16T13:00:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 79499264 bytes 2024-09-16T13:00:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 115673088 bytes 2024-09-16T13:00:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 172261376 bytes 2024-09-16T13:00:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 227174400 bytes 2024-09-16T13:00:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 261953536 bytes 2024-09-16T13:00:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 296821760 bytes 2024-09-16T13:00:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 334052352 bytes 2024-09-16T13:00:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 334653440 bytes 2024-09-16T13:00:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 380548096 bytes 2024-09-16T13:00:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 402966528 bytes 2024-09-16T13:00:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 494325760 bytes 2024-09-16T13:00:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 538941440 bytes 2024-09-16T13:00:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 575046656 bytes 2024-09-16T13:00:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 614559744 bytes 2024-09-16T13:00:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 675546112 bytes 2024-09-16T13:00:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 694556672 bytes 2024-09-16T13:00:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 702528512 bytes 2024-09-16T13:00:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 733235200 bytes 2024-09-16T13:00:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 747022336 bytes 2024-09-16T13:00:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 770690048 bytes 2024-09-16T13:00:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 824027136 bytes 2024-09-16T13:00:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 867599360 bytes 2024-09-16T13:00:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 900227072 bytes 2024-09-16T13:00:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 957511680 bytes 2024-09-16T13:00:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 984770560 bytes 2024-09-16T13:00:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1007880192 bytes 2024-09-16T13:00:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1026085888 bytes 2024-09-16T13:00:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1065047040 bytes 2024-09-16T13:00:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1097034752 bytes 2024-09-16T13:00:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1137495040 bytes 2024-09-16T13:00:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1167009792 bytes 2024-09-16T13:00:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1246934016 bytes 2024-09-16T13:00:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1282876416 bytes 2024-09-16T13:00:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1307916288 bytes 2024-09-16T13:00:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1330791424 bytes 2024-09-16T13:00:18.787 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T13:00:18.788 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:00:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:00:18.788 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T13:00:18.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T13:00:18.789 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-16T13:00:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-09-16T13:00:18.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T13:00:18.795 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-16T13:00:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-16T13:00:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_34': 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 142a9e2359ea 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.142a9e2359ea 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:18 2024 2024-09-16T13:00:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:18 2024 2024-09-16T13:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:18 2024 2024-09-16T13:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 321f4d24-c15c-4025-9b78-f2e2dd17716a 2024-09-16T13:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:00:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-16T13:00:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 1199104 bytes 2024-09-16T13:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 58559488 bytes 2024-09-16T13:00:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 126089216 bytes 2024-09-16T13:00:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 181449728 bytes 2024-09-16T13:00:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 217285632 bytes 2024-09-16T13:00:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 231663616 bytes 2024-09-16T13:00:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 250503168 bytes 2024-09-16T13:00:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 286685184 bytes 2024-09-16T13:00:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 378890240 bytes 2024-09-16T13:00:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 380124160 bytes 2024-09-16T13:00:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 432668672 bytes 2024-09-16T13:00:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 498882560 bytes 2024-09-16T13:00:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 530372608 bytes 2024-09-16T13:00:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 538642432 bytes 2024-09-16T13:00:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 572634112 bytes 2024-09-16T13:00:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 573235200 bytes 2024-09-16T13:00:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 613856256 bytes 2024-09-16T13:00:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 709455872 bytes 2024-09-16T13:00:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 751723520 bytes 2024-09-16T13:00:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 782855168 bytes 2024-09-16T13:00:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 783459328 bytes 2024-09-16T13:00:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 816718848 bytes 2024-09-16T13:00:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 823644160 bytes 2024-09-16T13:00:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 857995264 bytes 2024-09-16T13:00:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 859554816 bytes 2024-09-16T13:00:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 883860480 bytes 2024-09-16T13:00:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 917134336 bytes 2024-09-16T13:00:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 934649856 bytes 2024-09-16T13:00:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 942256128 bytes 2024-09-16T13:00:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1033750528 bytes 2024-09-16T13:00:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1051475968 bytes 2024-09-16T13:00:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1107354624 bytes 2024-09-16T13:00:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1107965952 bytes 2024-09-16T13:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1148368896 bytes 2024-09-16T13:00:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1210261504 bytes 2024-09-16T13:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1280603136 bytes 2024-09-16T13:00:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1312595968 bytes 2024-09-16T13:00:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1381038080 bytes 2024-09-16T13:00:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1423630336 bytes 2024-09-16T13:00:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1503540224 bytes 2024-09-16T13:00:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1541599232 bytes 2024-09-16T13:00:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1566477312 bytes 2024-09-16T13:00:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1601363968 bytes 2024-09-16T13:00:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1642687488 bytes 2024-09-16T13:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1673320448 bytes 2024-09-16T13:00:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1708541952 bytes 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-09-16T13:00:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-16T13:00:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:00:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-16T13:00:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-16T13:00:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_35': 2024-09-16T13:00:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 143fe65f91e5 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.143fe65f91e5 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:27 2024 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:27 2024 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:27 2024 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1bac1448-c694-4f80-90b6-6bd1f7068529 2024-09-16T13:00:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-16T13:00:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:00:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T13:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 57385984 bytes 2024-09-16T13:00:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 92827648 bytes 2024-09-16T13:00:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 162920448 bytes 2024-09-16T13:00:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 195862528 bytes 2024-09-16T13:00:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 271492096 bytes 2024-09-16T13:00:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 308746240 bytes 2024-09-16T13:00:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 349621248 bytes 2024-09-16T13:00:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 394261504 bytes 2024-09-16T13:00:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 394881024 bytes 2024-09-16T13:00:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 478406656 bytes 2024-09-16T13:00:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 531411968 bytes 2024-09-16T13:00:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 556861440 bytes 2024-09-16T13:00:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 580339712 bytes 2024-09-16T13:00:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 616982528 bytes 2024-09-16T13:00:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 641262592 bytes 2024-09-16T13:00:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 707406848 bytes 2024-09-16T13:00:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 709558272 bytes 2024-09-16T13:00:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 787424256 bytes 2024-09-16T13:00:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 867443712 bytes 2024-09-16T13:00:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 962921472 bytes 2024-09-16T13:00:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1010893824 bytes 2024-09-16T13:00:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1031747584 bytes 2024-09-16T13:00:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1032351744 bytes 2024-09-16T13:00:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1107615744 bytes 2024-09-16T13:00:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1194705920 bytes 2024-09-16T13:00:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1257596928 bytes 2024-09-16T13:00:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1298050048 bytes 2024-09-16T13:00:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1300689920 bytes 2024-09-16T13:00:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1377525760 bytes 2024-09-16T13:00:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1414792192 bytes 2024-09-16T13:00:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1453924352 bytes 2024-09-16T13:00:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1547725824 bytes 2024-09-16T13:00:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1589827584 bytes 2024-09-16T13:00:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1621065728 bytes 2024-09-16T13:00:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1648316416 bytes 2024-09-16T13:00:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1693918208 bytes 2024-09-16T13:00:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1760649216 bytes 2024-09-16T13:00:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1785073664 bytes 2024-09-16T13:00:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1844333568 bytes 2024-09-16T13:00:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1885594624 bytes 2024-09-16T13:00:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1890955264 bytes 2024-09-16T13:00:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1925002240 bytes 2024-09-16T13:00:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1942128640 bytes 2024-09-16T13:00:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1960192000 bytes 2024-09-16T13:00:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2003746816 bytes 2024-09-16T13:00:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2062735360 bytes 2024-09-16T13:00:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2127958016 bytes 2024-09-16T13:00:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2181481472 bytes 2024-09-16T13:00:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2256532480 bytes 2024-09-16T13:00:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2334431232 bytes 2024-09-16T13:00:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2367686656 bytes 2024-09-16T13:00:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2406776832 bytes 2024-09-16T13:00:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2423744512 bytes 2024-09-16T13:00:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-09-16T13:00:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-16T13:00:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:00:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-16T13:00:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-16T13:00:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_36': 2024-09-16T13:00:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 144bf9856143 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.144bf9856143 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:39 2024 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:39 2024 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:39 2024 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 573c8bbb-eb7d-47c7-a3f5-dd7fbc895516 2024-09-16T13:00:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-16T13:00:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:00:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T13:00:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 56433664 bytes 2024-09-16T13:00:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 125851648 bytes 2024-09-16T13:00:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 126466048 bytes 2024-09-16T13:00:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 143799296 bytes 2024-09-16T13:00:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 160854016 bytes 2024-09-16T13:00:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 214014976 bytes 2024-09-16T13:00:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 250273792 bytes 2024-09-16T13:00:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 269373440 bytes 2024-09-16T13:00:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 281284608 bytes 2024-09-16T13:00:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 281890816 bytes 2024-09-16T13:00:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 300038144 bytes 2024-09-16T13:00:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 334310400 bytes 2024-09-16T13:00:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 373319680 bytes 2024-09-16T13:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 411714560 bytes 2024-09-16T13:00:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 506171392 bytes 2024-09-16T13:00:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 558537728 bytes 2024-09-16T13:00:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 585975808 bytes 2024-09-16T13:00:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 604737536 bytes 2024-09-16T13:00:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 622695424 bytes 2024-09-16T13:00:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 658017280 bytes 2024-09-16T13:00:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 681988096 bytes 2024-09-16T13:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 730713088 bytes 2024-09-16T13:00:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 746734592 bytes 2024-09-16T13:00:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 770454528 bytes 2024-09-16T13:00:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 853057536 bytes 2024-09-16T13:00:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 978549760 bytes 2024-09-16T13:00:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1018344448 bytes 2024-09-16T13:00:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1072988160 bytes 2024-09-16T13:00:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1130507264 bytes 2024-09-16T13:00:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-16T13:00:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-16T13:00:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_37': 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14577de1115e 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14577de1115e 2024-09-16T13:00:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:45 2024 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:45 2024 2024-09-16T13:00:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:45 2024 2024-09-16T13:00:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: da41c0e4-7e5d-4612-8458-a799307ed96f 2024-09-16T13:00:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-16T13:00:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:00:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:00:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 68285440 bytes 2024-09-16T13:00:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 142856192 bytes 2024-09-16T13:00:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 167923712 bytes 2024-09-16T13:00:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 230524928 bytes 2024-09-16T13:00:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 254812160 bytes 2024-09-16T13:00:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 311919616 bytes 2024-09-16T13:00:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 312511488 bytes 2024-09-16T13:00:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 327009280 bytes 2024-09-16T13:00:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 393957376 bytes 2024-09-16T13:00:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 459009024 bytes 2024-09-16T13:00:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 512099328 bytes 2024-09-16T13:00:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 556950528 bytes 2024-09-16T13:00:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 626808832 bytes 2024-09-16T13:00:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 686931968 bytes 2024-09-16T13:00:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 694597632 bytes 2024-09-16T13:00:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 726753280 bytes 2024-09-16T13:00:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 765389824 bytes 2024-09-16T13:00:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 820189184 bytes 2024-09-16T13:00:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 890852352 bytes 2024-09-16T13:00:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 929178624 bytes 2024-09-16T13:00:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 961026048 bytes 2024-09-16T13:00:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1015405568 bytes 2024-09-16T13:00:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1072847872 bytes 2024-09-16T13:00:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1091260416 bytes 2024-09-16T13:00:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1118451712 bytes 2024-09-16T13:00:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1203222528 bytes 2024-09-16T13:00:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1203830784 bytes 2024-09-16T13:00:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1255761920 bytes 2024-09-16T13:00:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1328174080 bytes 2024-09-16T13:00:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1363015680 bytes 2024-09-16T13:00:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1384540160 bytes 2024-09-16T13:00:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1437648896 bytes 2024-09-16T13:00:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1486432256 bytes 2024-09-16T13:00:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1563305984 bytes 2024-09-16T13:00:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1590232064 bytes 2024-09-16T13:00:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1707953152 bytes 2024-09-16T13:00:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1726729216 bytes 2024-09-16T13:00:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1783577600 bytes 2024-09-16T13:00:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1822673920 bytes 2024-09-16T13:00:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1823299584 bytes 2024-09-16T13:00:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1831433216 bytes 2024-09-16T13:00:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1936044032 bytes 2024-09-16T13:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1999869952 bytes 2024-09-16T13:00:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2068361216 bytes 2024-09-16T13:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2103856128 bytes 2024-09-16T13:00:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2145827840 bytes 2024-09-16T13:00:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2187592704 bytes 2024-09-16T13:00:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2211791872 bytes 2024-09-16T13:00:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2218228736 bytes 2024-09-16T13:00:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2255679488 bytes 2024-09-16T13:00:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2310472704 bytes 2024-09-16T13:00:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2370778112 bytes 2024-09-16T13:00:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2426137600 bytes 2024-09-16T13:00:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2459630592 bytes 2024-09-16T13:00:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2473539584 bytes 2024-09-16T13:00:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2545376256 bytes 2024-09-16T13:00:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2600952832 bytes 2024-09-16T13:00:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2631778304 bytes 2024-09-16T13:00:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2670979072 bytes 2024-09-16T13:00:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2714170368 bytes 2024-09-16T13:00:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2770087936 bytes 2024-09-16T13:00:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2850072576 bytes 2024-09-16T13:00:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2894999552 bytes 2024-09-16T13:00:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2938471424 bytes 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:00:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:00:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:00:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:00:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-16T13:00:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:00:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-16T13:00:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_38': 2024-09-16T13:00:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:00:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:00:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1463e4a815b9 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1463e4a815b9 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:58 2024 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:58 2024 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:58 2024 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 721e34a7-a4b2-4332-a181-ccded18e8696 2024-09-16T13:00:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:00:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:00:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-16T13:00:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:00:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:00:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:00:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:00:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:00:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:00:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:00:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 23314432 bytes 2024-09-16T13:00:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 71699456 bytes 2024-09-16T13:00:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 89328640 bytes 2024-09-16T13:01:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 127715328 bytes 2024-09-16T13:01:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 128355328 bytes 2024-09-16T13:01:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 168974336 bytes 2024-09-16T13:01:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 255450112 bytes 2024-09-16T13:01:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 280896512 bytes 2024-09-16T13:01:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 281519104 bytes 2024-09-16T13:01:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 306136064 bytes 2024-09-16T13:01:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 306740224 bytes 2024-09-16T13:01:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 335708160 bytes 2024-09-16T13:01:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 370003968 bytes 2024-09-16T13:01:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 433049600 bytes 2024-09-16T13:01:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 461952000 bytes 2024-09-16T13:01:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 512932864 bytes 2024-09-16T13:01:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 569312256 bytes 2024-09-16T13:01:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 593814528 bytes 2024-09-16T13:01:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 657031168 bytes 2024-09-16T13:01:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 662435840 bytes 2024-09-16T13:01:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 684655616 bytes 2024-09-16T13:01:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 722387968 bytes 2024-09-16T13:01:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 736322560 bytes 2024-09-16T13:01:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 799160320 bytes 2024-09-16T13:01:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 839093248 bytes 2024-09-16T13:01:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 870727680 bytes 2024-09-16T13:01:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 902154240 bytes 2024-09-16T13:01:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 947356672 bytes 2024-09-16T13:01:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1016654848 bytes 2024-09-16T13:01:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1019486208 bytes 2024-09-16T13:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1053733888 bytes 2024-09-16T13:01:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1097585664 bytes 2024-09-16T13:01:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1145435136 bytes 2024-09-16T13:01:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1172557824 bytes 2024-09-16T13:01:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1179958272 bytes 2024-09-16T13:01:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1253410816 bytes 2024-09-16T13:01:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1314424832 bytes 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:01:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-16T13:01:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:01:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-16T13:01:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_39': 2024-09-16T13:01:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:01:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 146f1806a04e 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.146f1806a04e 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:05 2024 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:05 2024 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:05 2024 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:01:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:01:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: fe577bd1-b7b2-46e6-adee-f694b7fb7801 2024-09-16T13:01:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:01:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:01:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T13:01:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:01:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:01:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39649280 bytes 2024-09-16T13:01:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 62767104 bytes 2024-09-16T13:01:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 75445248 bytes 2024-09-16T13:01:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 130041856 bytes 2024-09-16T13:01:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 162679808 bytes 2024-09-16T13:01:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 195548160 bytes 2024-09-16T13:01:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 226817024 bytes 2024-09-16T13:01:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 303357952 bytes 2024-09-16T13:01:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 353964032 bytes 2024-09-16T13:01:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 379363328 bytes 2024-09-16T13:01:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 431905792 bytes 2024-09-16T13:01:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 475912192 bytes 2024-09-16T13:01:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 476519424 bytes 2024-09-16T13:01:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 502674432 bytes 2024-09-16T13:01:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 503291904 bytes 2024-09-16T13:01:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 561974272 bytes 2024-09-16T13:01:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 610516992 bytes 2024-09-16T13:01:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 704104448 bytes 2024-09-16T13:01:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 791647232 bytes 2024-09-16T13:01:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 827598848 bytes 2024-09-16T13:01:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 892359680 bytes 2024-09-16T13:01:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 912665600 bytes 2024-09-16T13:01:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 924352512 bytes 2024-09-16T13:01:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 964486144 bytes 2024-09-16T13:01:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 991850496 bytes 2024-09-16T13:01:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1011504128 bytes 2024-09-16T13:01:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1030443008 bytes 2024-09-16T13:01:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1086330880 bytes 2024-09-16T13:01:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1112757248 bytes 2024-09-16T13:01:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1149588480 bytes 2024-09-16T13:01:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1284722688 bytes 2024-09-16T13:01:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1317732352 bytes 2024-09-16T13:01:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1365851136 bytes 2024-09-16T13:01:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1366474752 bytes 2024-09-16T13:01:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1382976512 bytes 2024-09-16T13:01:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1469749248 bytes 2024-09-16T13:01:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1483542528 bytes 2024-09-16T13:01:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1543916544 bytes 2024-09-16T13:01:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1584489472 bytes 2024-09-16T13:01:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1586205696 bytes 2024-09-16T13:01:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1609774080 bytes 2024-09-16T13:01:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1694746624 bytes 2024-09-16T13:01:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1704962048 bytes 2024-09-16T13:01:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1794777088 bytes 2024-09-16T13:01:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1882575872 bytes 2024-09-16T13:01:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1924854784 bytes 2024-09-16T13:01:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1966020608 bytes 2024-09-16T13:01:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2007433216 bytes 2024-09-16T13:01:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2074225664 bytes 2024-09-16T13:01:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2099988480 bytes 2024-09-16T13:01:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2154468352 bytes 2024-09-16T13:01:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2195044352 bytes 2024-09-16T13:01:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2230122496 bytes 2024-09-16T13:01:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2256852992 bytes 2024-09-16T13:01:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2299127808 bytes 2024-09-16T13:01:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2364102656 bytes 2024-09-16T13:01:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2391329792 bytes 2024-09-16T13:01:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2405928960 bytes 2024-09-16T13:01:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2453449728 bytes 2024-09-16T13:01:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2469314560 bytes 2024-09-16T13:01:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2478377984 bytes 2024-09-16T13:01:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2569793536 bytes 2024-09-16T13:01:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2601621504 bytes 2024-09-16T13:01:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2613117952 bytes 2024-09-16T13:01:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2618265600 bytes 2024-09-16T13:01:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2719975424 bytes 2024-09-16T13:01:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2785736704 bytes 2024-09-16T13:01:17.856 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9602798562437757) > weight (0.25) 2024-09-16T13:01:17.856 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T13:01:17.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T13:01:17.856 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T13:01:17.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:01:17.856+0000 7f4cb6ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 38087) UID: 1000 2024-09-16T13:01:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2808756224 bytes 2024-09-16T13:01:18.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:01:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2825127936 bytes 2024-09-16T13:01:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2845643776 bytes 2024-09-16T13:01:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2945285120 bytes 2024-09-16T13:01:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3028285440 bytes 2024-09-16T13:01:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3076513792 bytes 2024-09-16T13:01:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3142543360 bytes 2024-09-16T13:01:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3212635136 bytes 2024-09-16T13:01:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3276282880 bytes 2024-09-16T13:01:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3380281344 bytes 2024-09-16T13:01:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3422482432 bytes 2024-09-16T13:01:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-16T13:01:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:01:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_40': 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 147b3fcc7f81 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.147b3fcc7f81 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:01:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:20 2024 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:20 2024 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:20 2024 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a44aec3e-055a-42d9-a75a-50b8b1713251 2024-09-16T13:01:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:01:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:01:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-16T13:01:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:01:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:01:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 44933120 bytes 2024-09-16T13:01:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 96064512 bytes 2024-09-16T13:01:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 129009664 bytes 2024-09-16T13:01:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 179166208 bytes 2024-09-16T13:01:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 189664256 bytes 2024-09-16T13:01:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 203767808 bytes 2024-09-16T13:01:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 218682368 bytes 2024-09-16T13:01:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 229892096 bytes 2024-09-16T13:01:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 236056576 bytes 2024-09-16T13:01:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 285119488 bytes 2024-09-16T13:01:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 338461696 bytes 2024-09-16T13:01:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 396277760 bytes 2024-09-16T13:01:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 436091904 bytes 2024-09-16T13:01:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 449998848 bytes 2024-09-16T13:01:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 450608128 bytes 2024-09-16T13:01:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 526389248 bytes 2024-09-16T13:01:23.857 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T13:01:23.857 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:01:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:01:23.858 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T13:01:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T13:01:23.858 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T13:01:23.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T13:01:23.861 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-16T13:01:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 566632448 bytes 2024-09-16T13:01:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 657354752 bytes 2024-09-16T13:01:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 723178496 bytes 2024-09-16T13:01:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 774900736 bytes 2024-09-16T13:01:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 806891520 bytes 2024-09-16T13:01:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 856819712 bytes 2024-09-16T13:01:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 880157696 bytes 2024-09-16T13:01:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 963233792 bytes 2024-09-16T13:01:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1018928128 bytes 2024-09-16T13:01:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1040339968 bytes 2024-09-16T13:01:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1097680896 bytes 2024-09-16T13:01:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1145272320 bytes 2024-09-16T13:01:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1196668928 bytes 2024-09-16T13:01:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1239734272 bytes 2024-09-16T13:01:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1265103872 bytes 2024-09-16T13:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1270539264 bytes 2024-09-16T13:01:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1329802240 bytes 2024-09-16T13:01:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1354913792 bytes 2024-09-16T13:01:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1391225856 bytes 2024-09-16T13:01:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1420114944 bytes 2024-09-16T13:01:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1488064512 bytes 2024-09-16T13:01:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1515645952 bytes 2024-09-16T13:01:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1592404992 bytes 2024-09-16T13:01:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1605026816 bytes 2024-09-16T13:01:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1626330112 bytes 2024-09-16T13:01:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1710030848 bytes 2024-09-16T13:01:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1769328640 bytes 2024-09-16T13:01:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1808786432 bytes 2024-09-16T13:01:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1832723456 bytes 2024-09-16T13:01:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1863791616 bytes 2024-09-16T13:01:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1883307008 bytes 2024-09-16T13:01:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1925084160 bytes 2024-09-16T13:01:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1998098432 bytes 2024-09-16T13:01:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2083727360 bytes 2024-09-16T13:01:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:01:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-16T13:01:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:01:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_41': 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 148b94b4895f 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.148b94b4895f 2024-09-16T13:01:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:30 2024 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:30 2024 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:30 2024 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3e6603df-4a85-45e2-92bc-3718de0d61a1 2024-09-16T13:01:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:01:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:01:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T13:01:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:01:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:01:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 15666176 bytes 2024-09-16T13:01:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 40223744 bytes 2024-09-16T13:01:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 85873664 bytes 2024-09-16T13:01:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 138396672 bytes 2024-09-16T13:01:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 175991808 bytes 2024-09-16T13:01:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 212356096 bytes 2024-09-16T13:01:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 282485760 bytes 2024-09-16T13:01:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 319738880 bytes 2024-09-16T13:01:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 382828544 bytes 2024-09-16T13:01:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 383432704 bytes 2024-09-16T13:01:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 438119424 bytes 2024-09-16T13:01:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 462950400 bytes 2024-09-16T13:01:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 567576576 bytes 2024-09-16T13:01:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 588401664 bytes 2024-09-16T13:01:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 626741248 bytes 2024-09-16T13:01:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 664120320 bytes 2024-09-16T13:01:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 689793024 bytes 2024-09-16T13:01:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 736647168 bytes 2024-09-16T13:01:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 770723840 bytes 2024-09-16T13:01:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 841583616 bytes 2024-09-16T13:01:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 869101568 bytes 2024-09-16T13:01:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 946183168 bytes 2024-09-16T13:01:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 986508288 bytes 2024-09-16T13:01:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1045983232 bytes 2024-09-16T13:01:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1084685312 bytes 2024-09-16T13:01:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1122867200 bytes 2024-09-16T13:01:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1135074304 bytes 2024-09-16T13:01:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1144689664 bytes 2024-09-16T13:01:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1187494912 bytes 2024-09-16T13:01:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:01:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-16T13:01:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:01:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-16T13:01:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_42': 2024-09-16T13:01:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:01:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:01:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1493cbf8fafe 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1493cbf8fafe 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:36 2024 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:36 2024 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:36 2024 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 362a7f1e-b75e-484c-bd35-6308692dbcfe 2024-09-16T13:01:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:01:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:01:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-16T13:01:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:01:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:01:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:01:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:01:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:01:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 95681536 bytes 2024-09-16T13:01:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 151725056 bytes 2024-09-16T13:01:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 193314816 bytes 2024-09-16T13:01:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 220548096 bytes 2024-09-16T13:01:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 294091776 bytes 2024-09-16T13:01:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 352158720 bytes 2024-09-16T13:01:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 365775872 bytes 2024-09-16T13:01:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 421397504 bytes 2024-09-16T13:01:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 455444480 bytes 2024-09-16T13:01:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 494450688 bytes 2024-09-16T13:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 509379584 bytes 2024-09-16T13:01:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 555841536 bytes 2024-09-16T13:01:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 581544960 bytes 2024-09-16T13:01:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 643480576 bytes 2024-09-16T13:01:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 694240256 bytes 2024-09-16T13:01:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 745264128 bytes 2024-09-16T13:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 778117120 bytes 2024-09-16T13:01:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 807054336 bytes 2024-09-16T13:01:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 809070592 bytes 2024-09-16T13:01:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 830166016 bytes 2024-09-16T13:01:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 849438720 bytes 2024-09-16T13:01:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 864727040 bytes 2024-09-16T13:01:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 942705664 bytes 2024-09-16T13:01:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 963133440 bytes 2024-09-16T13:01:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1011567616 bytes 2024-09-16T13:01:41.796 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.572858012064789) > weight (0.25) 2024-09-16T13:01:41.796 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6602264623660116) > weight (0.25) 2024-09-16T13:01:41.796 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9994829226828703) > weight (0.25) 2024-09-16T13:01:41.796 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T13:01:41.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T13:01:41.797 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T13:01:41.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:01:41.796+0000 7f7869ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 42131) UID: 1000 2024-09-16T13:01:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1056497664 bytes 2024-09-16T13:01:41.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:01:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1112670208 bytes 2024-09-16T13:01:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1163329536 bytes 2024-09-16T13:01:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1249856512 bytes 2024-09-16T13:01:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1284666368 bytes 2024-09-16T13:01:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1327364096 bytes 2024-09-16T13:01:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1351991296 bytes 2024-09-16T13:01:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1389815808 bytes 2024-09-16T13:01:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1392013312 bytes 2024-09-16T13:01:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1425826816 bytes 2024-09-16T13:01:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1479726080 bytes 2024-09-16T13:01:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1514872832 bytes 2024-09-16T13:01:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1630827520 bytes 2024-09-16T13:01:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1668477952 bytes 2024-09-16T13:01:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1713245184 bytes 2024-09-16T13:01:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1749307392 bytes 2024-09-16T13:01:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1851209728 bytes 2024-09-16T13:01:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1880384512 bytes 2024-09-16T13:01:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1927589888 bytes 2024-09-16T13:01:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1979068416 bytes 2024-09-16T13:01:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1992458240 bytes 2024-09-16T13:01:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2036913152 bytes 2024-09-16T13:01:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2082371584 bytes 2024-09-16T13:01:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2176801792 bytes 2024-09-16T13:01:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2218284032 bytes 2024-09-16T13:01:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2265361408 bytes 2024-09-16T13:01:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2297544704 bytes 2024-09-16T13:01:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2359853056 bytes 2024-09-16T13:01:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2374753280 bytes 2024-09-16T13:01:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2398571520 bytes 2024-09-16T13:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2415492096 bytes 2024-09-16T13:01:47.420 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T13:01:47.417+0000 7f9655eab640 -1 reset not still connected to 0x5631195b8820 2024-09-16T13:01:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2421043200 bytes 2024-09-16T13:01:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2433745920 bytes 2024-09-16T13:01:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2469975040 bytes 2024-09-16T13:01:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2535822336 bytes 2024-09-16T13:01:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2581502976 bytes 2024-09-16T13:01:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2629432320 bytes 2024-09-16T13:01:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2694323200 bytes 2024-09-16T13:01:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2738135040 bytes 2024-09-16T13:01:48.800 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T13:01:48.800 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:01:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:01:48.800 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T13:01:48.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T13:01:48.801 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T13:01:48.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T13:01:48.804 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-16T13:01:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2759277568 bytes 2024-09-16T13:01:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2765024256 bytes 2024-09-16T13:01:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2765617152 bytes 2024-09-16T13:01:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2816547840 bytes 2024-09-16T13:01:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:01:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:01:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:01:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-16T13:01:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:01:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_43': 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14a3de54327b 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14a3de54327b 2024-09-16T13:01:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:49 2024 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:49 2024 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:49 2024 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 89399335-667d-40ff-83af-2f8d80ec3800 2024-09-16T13:01:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:01:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:01:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:01:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:01:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:01:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 90884096 bytes 2024-09-16T13:01:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 95323136 bytes 2024-09-16T13:01:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 143653888 bytes 2024-09-16T13:01:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 220584960 bytes 2024-09-16T13:01:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 231339008 bytes 2024-09-16T13:01:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 270343168 bytes 2024-09-16T13:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 292758528 bytes 2024-09-16T13:01:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 327566336 bytes 2024-09-16T13:01:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 353508352 bytes 2024-09-16T13:01:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 355721216 bytes 2024-09-16T13:01:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 410517504 bytes 2024-09-16T13:01:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 497931264 bytes 2024-09-16T13:01:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 558067712 bytes 2024-09-16T13:01:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 611717120 bytes 2024-09-16T13:01:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 627134464 bytes 2024-09-16T13:01:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 649305088 bytes 2024-09-16T13:01:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 669564928 bytes 2024-09-16T13:01:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 723829760 bytes 2024-09-16T13:01:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 783972352 bytes 2024-09-16T13:01:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 839272448 bytes 2024-09-16T13:01:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 840023040 bytes 2024-09-16T13:01:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 874019840 bytes 2024-09-16T13:01:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 926644224 bytes 2024-09-16T13:01:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 970689536 bytes 2024-09-16T13:01:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 992114688 bytes 2024-09-16T13:01:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 992719872 bytes 2024-09-16T13:01:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1016843264 bytes 2024-09-16T13:01:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1041036288 bytes 2024-09-16T13:01:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1060314112 bytes 2024-09-16T13:01:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1110483968 bytes 2024-09-16T13:01:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1199784960 bytes 2024-09-16T13:01:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1252686848 bytes 2024-09-16T13:01:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1308818432 bytes 2024-09-16T13:01:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1374374912 bytes 2024-09-16T13:01:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1436408832 bytes 2024-09-16T13:01:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1465220096 bytes 2024-09-16T13:01:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1509187584 bytes 2024-09-16T13:01:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1590627328 bytes 2024-09-16T13:01:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1607591936 bytes 2024-09-16T13:01:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1608185856 bytes 2024-09-16T13:01:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1694189568 bytes 2024-09-16T13:01:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1733265408 bytes 2024-09-16T13:01:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1762363392 bytes 2024-09-16T13:01:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1776685056 bytes 2024-09-16T13:01:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1818416128 bytes 2024-09-16T13:01:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1844534272 bytes 2024-09-16T13:01:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1879862272 bytes 2024-09-16T13:01:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1937078272 bytes 2024-09-16T13:01:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1980529664 bytes 2024-09-16T13:01:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2019344384 bytes 2024-09-16T13:01:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2044441600 bytes 2024-09-16T13:01:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2065346560 bytes 2024-09-16T13:01:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2090015744 bytes 2024-09-16T13:01:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2128622592 bytes 2024-09-16T13:01:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2172028928 bytes 2024-09-16T13:01:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2176866304 bytes 2024-09-16T13:01:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2182840320 bytes 2024-09-16T13:01:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2227689472 bytes 2024-09-16T13:02:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2274163712 bytes 2024-09-16T13:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2306920448 bytes 2024-09-16T13:02:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2347479040 bytes 2024-09-16T13:02:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2351749120 bytes 2024-09-16T13:02:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2371997696 bytes 2024-09-16T13:02:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2423144448 bytes 2024-09-16T13:02:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2447508480 bytes 2024-09-16T13:02:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2461316096 bytes 2024-09-16T13:02:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2498399232 bytes 2024-09-16T13:02:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2602200064 bytes 2024-09-16T13:02:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2636053504 bytes 2024-09-16T13:02:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2653882368 bytes 2024-09-16T13:02:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2687966208 bytes 2024-09-16T13:02:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2688552960 bytes 2024-09-16T13:02:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2712647680 bytes 2024-09-16T13:02:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2748070912 bytes 2024-09-16T13:02:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2771378176 bytes 2024-09-16T13:02:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-16T13:02:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_44': 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14adf97bc4ab 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14adf97bc4ab 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:02 2024 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:02 2024 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:02 2024 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 07e9e1a4-7494-4a2b-8016-b9a4a688b970 2024-09-16T13:02:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:02:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T13:02:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 62991360 bytes 2024-09-16T13:02:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 153121792 bytes 2024-09-16T13:02:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 203126784 bytes 2024-09-16T13:02:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 273096704 bytes 2024-09-16T13:02:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 301577216 bytes 2024-09-16T13:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 337629184 bytes 2024-09-16T13:02:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 380143616 bytes 2024-09-16T13:02:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 431104000 bytes 2024-09-16T13:02:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 473241600 bytes 2024-09-16T13:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 520749056 bytes 2024-09-16T13:02:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 560662528 bytes 2024-09-16T13:02:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 561273856 bytes 2024-09-16T13:02:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 630998016 bytes 2024-09-16T13:02:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 682107904 bytes 2024-09-16T13:02:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 765115392 bytes 2024-09-16T13:02:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 807463936 bytes 2024-09-16T13:02:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 957278208 bytes 2024-09-16T13:02:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 985631744 bytes 2024-09-16T13:02:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1115121664 bytes 2024-09-16T13:02:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1200501760 bytes 2024-09-16T13:02:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1204110336 bytes 2024-09-16T13:02:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1230171136 bytes 2024-09-16T13:02:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1248943104 bytes 2024-09-16T13:02:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1295860736 bytes 2024-09-16T13:02:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1332957184 bytes 2024-09-16T13:02:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1388732416 bytes 2024-09-16T13:02:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1427888128 bytes 2024-09-16T13:02:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1488945152 bytes 2024-09-16T13:02:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1512027136 bytes 2024-09-16T13:02:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1588694016 bytes 2024-09-16T13:02:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1625432064 bytes 2024-09-16T13:02:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1632128000 bytes 2024-09-16T13:02:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1636629504 bytes 2024-09-16T13:02:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1653248000 bytes 2024-09-16T13:02:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1682394112 bytes 2024-09-16T13:02:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1714481152 bytes 2024-09-16T13:02:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1792548864 bytes 2024-09-16T13:02:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1804752896 bytes 2024-09-16T13:02:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1876501504 bytes 2024-09-16T13:02:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1915214848 bytes 2024-09-16T13:02:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1963314176 bytes 2024-09-16T13:02:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2000061440 bytes 2024-09-16T13:02:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2027464704 bytes 2024-09-16T13:02:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2039288832 bytes 2024-09-16T13:02:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2066937856 bytes 2024-09-16T13:02:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2143365120 bytes 2024-09-16T13:02:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2201147392 bytes 2024-09-16T13:02:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2298681344 bytes 2024-09-16T13:02:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2359624704 bytes 2024-09-16T13:02:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2391365632 bytes 2024-09-16T13:02:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2438524928 bytes 2024-09-16T13:02:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2449257472 bytes 2024-09-16T13:02:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2505698304 bytes 2024-09-16T13:02:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2585102336 bytes 2024-09-16T13:02:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2628188160 bytes 2024-09-16T13:02:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2680077312 bytes 2024-09-16T13:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2681893888 bytes 2024-09-16T13:02:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2732149760 bytes 2024-09-16T13:02:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2801321984 bytes 2024-09-16T13:02:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2854561792 bytes 2024-09-16T13:02:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2867750912 bytes 2024-09-16T13:02:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2888306688 bytes 2024-09-16T13:02:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2944710656 bytes 2024-09-16T13:02:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2999291904 bytes 2024-09-16T13:02:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3049080832 bytes 2024-09-16T13:02:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3105119232 bytes 2024-09-16T13:02:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3180112896 bytes 2024-09-16T13:02:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3180730368 bytes 2024-09-16T13:02:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3227750400 bytes 2024-09-16T13:02:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3249500160 bytes 2024-09-16T13:02:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3294988288 bytes 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-16T13:02:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-09-16T13:02:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-16T13:02:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:02:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-16T13:02:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_45': 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14b66c30403f 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14b66c30403f 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:16 2024 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:16 2024 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:16 2024 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 65c4a20f-84d6-4179-a149-1fd4a4176f5d 2024-09-16T13:02:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-16T13:02:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:02:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-16T13:02:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 26543104 bytes 2024-09-16T13:02:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 50751488 bytes 2024-09-16T13:02:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 79130624 bytes 2024-09-16T13:02:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 80916480 bytes 2024-09-16T13:02:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 114268160 bytes 2024-09-16T13:02:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 185221120 bytes 2024-09-16T13:02:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 202802176 bytes 2024-09-16T13:02:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 234935296 bytes 2024-09-16T13:02:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 262096896 bytes 2024-09-16T13:02:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 339172352 bytes 2024-09-16T13:02:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 363661312 bytes 2024-09-16T13:02:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 427677696 bytes 2024-09-16T13:02:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 473171968 bytes 2024-09-16T13:02:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 484323328 bytes 2024-09-16T13:02:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 500619264 bytes 2024-09-16T13:02:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 553296896 bytes 2024-09-16T13:02:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 582672384 bytes 2024-09-16T13:02:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 605065216 bytes 2024-09-16T13:02:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 702612480 bytes 2024-09-16T13:02:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 735416320 bytes 2024-09-16T13:02:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 778224640 bytes 2024-09-16T13:02:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 815792128 bytes 2024-09-16T13:02:21.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 822738944 bytes 2024-09-16T13:02:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 869714944 bytes 2024-09-16T13:02:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 906948608 bytes 2024-09-16T13:02:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 970217472 bytes 2024-09-16T13:02:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 997346304 bytes 2024-09-16T13:02:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1025212416 bytes 2024-09-16T13:02:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1102834688 bytes 2024-09-16T13:02:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1178733568 bytes 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-16T13:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-09-16T13:02:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-16T13:02:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_46': 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14bfad6c2f98 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14bfad6c2f98 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:22 2024 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:22 2024 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:22 2024 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d0e5b05e-776b-457f-b59b-3da9cd242b9e 2024-09-16T13:02:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-16T13:02:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:02:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:02:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39939072 bytes 2024-09-16T13:02:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 42550272 bytes 2024-09-16T13:02:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 77969408 bytes 2024-09-16T13:02:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 150214656 bytes 2024-09-16T13:02:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 152710144 bytes 2024-09-16T13:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 187883520 bytes 2024-09-16T13:02:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 228182016 bytes 2024-09-16T13:02:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 239112192 bytes 2024-09-16T13:02:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 242507776 bytes 2024-09-16T13:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 285953024 bytes 2024-09-16T13:02:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 333665280 bytes 2024-09-16T13:02:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 336122880 bytes 2024-09-16T13:02:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 373475328 bytes 2024-09-16T13:02:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 414892032 bytes 2024-09-16T13:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 426499072 bytes 2024-09-16T13:02:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 509435904 bytes 2024-09-16T13:02:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 550398976 bytes 2024-09-16T13:02:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 602488832 bytes 2024-09-16T13:02:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 603313152 bytes 2024-09-16T13:02:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 618159104 bytes 2024-09-16T13:02:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 699301888 bytes 2024-09-16T13:02:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 703184896 bytes 2024-09-16T13:02:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 784524288 bytes 2024-09-16T13:02:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 881282048 bytes 2024-09-16T13:02:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 938049536 bytes 2024-09-16T13:02:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 952172544 bytes 2024-09-16T13:02:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 982767616 bytes 2024-09-16T13:02:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 998649856 bytes 2024-09-16T13:02:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1037714432 bytes 2024-09-16T13:02:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1052571648 bytes 2024-09-16T13:02:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1113404416 bytes 2024-09-16T13:02:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1150744576 bytes 2024-09-16T13:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1250129920 bytes 2024-09-16T13:02:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1250756608 bytes 2024-09-16T13:02:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1350175744 bytes 2024-09-16T13:02:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1371483136 bytes 2024-09-16T13:02:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1372098560 bytes 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-09-16T13:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-16T13:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:02:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-16T13:02:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_47': 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14c76c35625 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14c76c35625 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:29 2024 2024-09-16T13:02:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:29 2024 2024-09-16T13:02:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:29 2024 2024-09-16T13:02:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8cae9722-25fd-42a4-9bfa-950a5f46d236 2024-09-16T13:02:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:02:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T13:02:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 51380224 bytes 2024-09-16T13:02:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 67918848 bytes 2024-09-16T13:02:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 103942144 bytes 2024-09-16T13:02:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 169270272 bytes 2024-09-16T13:02:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 197440512 bytes 2024-09-16T13:02:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 254990336 bytes 2024-09-16T13:02:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 301724672 bytes 2024-09-16T13:02:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 393181184 bytes 2024-09-16T13:02:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 395511808 bytes 2024-09-16T13:02:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 452505600 bytes 2024-09-16T13:02:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 495439872 bytes 2024-09-16T13:02:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 514696192 bytes 2024-09-16T13:02:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 592528384 bytes 2024-09-16T13:02:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 634710016 bytes 2024-09-16T13:02:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 707221504 bytes 2024-09-16T13:02:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 770168832 bytes 2024-09-16T13:02:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 798431232 bytes 2024-09-16T13:02:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 826461184 bytes 2024-09-16T13:02:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 881636352 bytes 2024-09-16T13:02:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 906368000 bytes 2024-09-16T13:02:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 920887296 bytes 2024-09-16T13:02:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 969000960 bytes 2024-09-16T13:02:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 979163136 bytes 2024-09-16T13:02:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1020141568 bytes 2024-09-16T13:02:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1084803072 bytes 2024-09-16T13:02:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1108727808 bytes 2024-09-16T13:02:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1142341632 bytes 2024-09-16T13:02:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1197817856 bytes 2024-09-16T13:02:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1268662272 bytes 2024-09-16T13:02:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1307701248 bytes 2024-09-16T13:02:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1345966080 bytes 2024-09-16T13:02:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1401728000 bytes 2024-09-16T13:02:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1424388096 bytes 2024-09-16T13:02:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1433149440 bytes 2024-09-16T13:02:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1437930496 bytes 2024-09-16T13:02:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1517463552 bytes 2024-09-16T13:02:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1542763520 bytes 2024-09-16T13:02:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1597498368 bytes 2024-09-16T13:02:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1623358464 bytes 2024-09-16T13:02:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1642295296 bytes 2024-09-16T13:02:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1731740672 bytes 2024-09-16T13:02:37.862 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8363845240433186) > weight (0.25) 2024-09-16T13:02:37.862 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T13:02:37.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T13:02:37.863 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T13:02:37.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:02:37.861+0000 7fd1f51f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 50923) UID: 1000 2024-09-16T13:02:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1777546240 bytes 2024-09-16T13:02:38.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:02:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1819737088 bytes 2024-09-16T13:02:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1946856448 bytes 2024-09-16T13:02:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1982857216 bytes 2024-09-16T13:02:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2029241344 bytes 2024-09-16T13:02:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2050355200 bytes 2024-09-16T13:02:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2145737728 bytes 2024-09-16T13:02:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2201623552 bytes 2024-09-16T13:02:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2235897856 bytes 2024-09-16T13:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2261949440 bytes 2024-09-16T13:02:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2337220608 bytes 2024-09-16T13:02:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2369913856 bytes 2024-09-16T13:02:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2403812352 bytes 2024-09-16T13:02:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2436178944 bytes 2024-09-16T13:02:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2473435136 bytes 2024-09-16T13:02:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2495469568 bytes 2024-09-16T13:02:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2532323328 bytes 2024-09-16T13:02:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2663582720 bytes 2024-09-16T13:02:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2667960320 bytes 2024-09-16T13:02:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2714875904 bytes 2024-09-16T13:02:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2787136512 bytes 2024-09-16T13:02:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2838697984 bytes 2024-09-16T13:02:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2868538368 bytes 2024-09-16T13:02:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2905064448 bytes 2024-09-16T13:02:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2985254912 bytes 2024-09-16T13:02:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3034875904 bytes 2024-09-16T13:02:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3076962304 bytes 2024-09-16T13:02:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3134181376 bytes 2024-09-16T13:02:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3134776320 bytes 2024-09-16T13:02:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3161977856 bytes 2024-09-16T13:02:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3258543104 bytes 2024-09-16T13:02:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3329958912 bytes 2024-09-16T13:02:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3330579456 bytes 2024-09-16T13:02:43.864 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T13:02:43.864 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:02:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:02:43.865 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T13:02:43.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T13:02:43.865 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T13:02:43.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T13:02:43.869 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-16T13:02:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3469096960 bytes 2024-09-16T13:02:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3469705216 bytes 2024-09-16T13:02:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3504747520 bytes 2024-09-16T13:02:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3521994752 bytes 2024-09-16T13:02:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-09-16T13:02:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-16T13:02:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:02:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-16T13:02:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-16T13:02:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_48': 2024-09-16T13:02:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14d484b29b48 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14d484b29b48 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:44 2024 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:44 2024 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:44 2024 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: b0393f6b-edbc-4675-be86-563550c2a642 2024-09-16T13:02:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:02:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-16T13:02:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 71512064 bytes 2024-09-16T13:02:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 72129536 bytes 2024-09-16T13:02:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 116865024 bytes 2024-09-16T13:02:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 158400512 bytes 2024-09-16T13:02:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 193728512 bytes 2024-09-16T13:02:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 218822656 bytes 2024-09-16T13:02:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 294506496 bytes 2024-09-16T13:02:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 315352064 bytes 2024-09-16T13:02:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 373368832 bytes 2024-09-16T13:02:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 379719680 bytes 2024-09-16T13:02:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 424182784 bytes 2024-09-16T13:02:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 488290304 bytes 2024-09-16T13:02:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 527486976 bytes 2024-09-16T13:02:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 549692416 bytes 2024-09-16T13:02:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 602755072 bytes 2024-09-16T13:02:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 671716352 bytes 2024-09-16T13:02:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 672340992 bytes 2024-09-16T13:02:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 747536384 bytes 2024-09-16T13:02:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 815640576 bytes 2024-09-16T13:02:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 835473408 bytes 2024-09-16T13:02:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 858827776 bytes 2024-09-16T13:02:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 887259136 bytes 2024-09-16T13:02:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 922668032 bytes 2024-09-16T13:02:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 967556096 bytes 2024-09-16T13:02:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 980451328 bytes 2024-09-16T13:02:49.982 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:02:49.979+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dd16b790 2024-09-16T13:02:49.984 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:02:49.981+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e98db6c0 2024-09-16T13:02:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1053699072 bytes 2024-09-16T13:02:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1103134720 bytes 2024-09-16T13:02:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1171418112 bytes 2024-09-16T13:02:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1262109696 bytes 2024-09-16T13:02:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1287714816 bytes 2024-09-16T13:02:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1364177920 bytes 2024-09-16T13:02:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1395712000 bytes 2024-09-16T13:02:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1408438272 bytes 2024-09-16T13:02:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1437643776 bytes 2024-09-16T13:02:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1506434048 bytes 2024-09-16T13:02:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1589977088 bytes 2024-09-16T13:02:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1622892544 bytes 2024-09-16T13:02:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1659536384 bytes 2024-09-16T13:02:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1687445504 bytes 2024-09-16T13:02:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1752307712 bytes 2024-09-16T13:02:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1815302144 bytes 2024-09-16T13:02:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1828590592 bytes 2024-09-16T13:02:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1929227264 bytes 2024-09-16T13:02:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1998794752 bytes 2024-09-16T13:02:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2022041600 bytes 2024-09-16T13:02:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2068345856 bytes 2024-09-16T13:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2138185728 bytes 2024-09-16T13:02:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2198899712 bytes 2024-09-16T13:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2251285504 bytes 2024-09-16T13:02:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2288302080 bytes 2024-09-16T13:02:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2312070144 bytes 2024-09-16T13:02:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2381719552 bytes 2024-09-16T13:02:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2430584832 bytes 2024-09-16T13:02:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2472846336 bytes 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-16T13:02:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:02:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:02:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-16T13:02:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:02:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_49': 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14dc2c497095 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14dc2c497095 2024-09-16T13:02:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:55 2024 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:55 2024 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:55 2024 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7e4e667f-21c3-4325-899a-b12ec27f6f23 2024-09-16T13:02:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:02:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:02:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-16T13:02:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:02:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:02:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:02:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T13:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:02:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:02:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 52439040 bytes 2024-09-16T13:02:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 87983104 bytes 2024-09-16T13:02:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 98584576 bytes 2024-09-16T13:02:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 145860608 bytes 2024-09-16T13:02:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 223809536 bytes 2024-09-16T13:02:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 306493440 bytes 2024-09-16T13:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 349168640 bytes 2024-09-16T13:02:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 400654336 bytes 2024-09-16T13:02:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 432933888 bytes 2024-09-16T13:02:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 484247552 bytes 2024-09-16T13:02:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 500078592 bytes 2024-09-16T13:02:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 573468672 bytes 2024-09-16T13:02:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 617088000 bytes 2024-09-16T13:02:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 659600384 bytes 2024-09-16T13:02:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 690641920 bytes 2024-09-16T13:02:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 691243008 bytes 2024-09-16T13:02:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 711044096 bytes 2024-09-16T13:03:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 739953664 bytes 2024-09-16T13:03:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 755333120 bytes 2024-09-16T13:03:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 781908992 bytes 2024-09-16T13:03:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 882877440 bytes 2024-09-16T13:03:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 935902208 bytes 2024-09-16T13:03:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 967290880 bytes 2024-09-16T13:03:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 999801856 bytes 2024-09-16T13:03:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1020818432 bytes 2024-09-16T13:03:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1063586816 bytes 2024-09-16T13:03:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1106294784 bytes 2024-09-16T13:03:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1115918336 bytes 2024-09-16T13:03:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1166558208 bytes 2024-09-16T13:03:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1272249344 bytes 2024-09-16T13:03:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1345618944 bytes 2024-09-16T13:03:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1353415680 bytes 2024-09-16T13:03:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1383702528 bytes 2024-09-16T13:03:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1462502400 bytes 2024-09-16T13:03:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1468269568 bytes 2024-09-16T13:03:03.080 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T13:03:03.077+0000 7f09dd39d640 -1 reset not still connected to 0x558b1a003790 2024-09-16T13:03:03.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1544810496 bytes 2024-09-16T13:03:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1594657792 bytes 2024-09-16T13:03:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1648490496 bytes 2024-09-16T13:03:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1671450624 bytes 2024-09-16T13:03:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1703434240 bytes 2024-09-16T13:03:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1728489472 bytes 2024-09-16T13:03:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1789132800 bytes 2024-09-16T13:03:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1823372288 bytes 2024-09-16T13:03:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1869894656 bytes 2024-09-16T13:03:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1887046656 bytes 2024-09-16T13:03:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1932694528 bytes 2024-09-16T13:03:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1996899328 bytes 2024-09-16T13:03:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2083136512 bytes 2024-09-16T13:03:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2120462336 bytes 2024-09-16T13:03:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2163348480 bytes 2024-09-16T13:03:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2201137152 bytes 2024-09-16T13:03:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2243538944 bytes 2024-09-16T13:03:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2277197824 bytes 2024-09-16T13:03:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2291031040 bytes 2024-09-16T13:03:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2323708928 bytes 2024-09-16T13:03:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2418437120 bytes 2024-09-16T13:03:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2450292736 bytes 2024-09-16T13:03:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2492588032 bytes 2024-09-16T13:03:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2538465280 bytes 2024-09-16T13:03:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2562611200 bytes 2024-09-16T13:03:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2576862208 bytes 2024-09-16T13:03:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2604592128 bytes 2024-09-16T13:03:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2643835904 bytes 2024-09-16T13:03:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2676877312 bytes 2024-09-16T13:03:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2751726592 bytes 2024-09-16T13:03:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2775799808 bytes 2024-09-16T13:03:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2789954560 bytes 2024-09-16T13:03:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2816971776 bytes 2024-09-16T13:03:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2873380864 bytes 2024-09-16T13:03:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2880079872 bytes 2024-09-16T13:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2897672192 bytes 2024-09-16T13:03:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2912918528 bytes 2024-09-16T13:03:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2973914112 bytes 2024-09-16T13:03:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2979383296 bytes 2024-09-16T13:03:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3075333120 bytes 2024-09-16T13:03:09.805 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3464070494342639) > weight (0.25) 2024-09-16T13:03:09.805 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.27994897842700617) > weight (0.25) 2024-09-16T13:03:09.805 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.42579843242946447) > weight (0.25) 2024-09-16T13:03:09.805 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.38515409329068406) > weight (0.25) 2024-09-16T13:03:09.805 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T13:03:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:03:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:03:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:03:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:03:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-09-16T13:03:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-16T13:03:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-16T13:03:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:03:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:03:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-16T13:03:09.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:03:09.891+0000 7f8ad7dde640 -1 rbd::mirror::ImageReplayer: 0x55d9e5de5680 [3/07e9e1a4-7494-4a2b-8016-b9a4a688b970] start: already running 2024-09-16T13:03:09.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:03:09.896+0000 7f8ad95e1640 -1 rbd::mirror::ImageReplayer: 0x55d9e8562000 [3/3b3040ad-963e-4bf9-8b3f-18deb1f76778] start: already running 2024-09-16T13:03:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:03:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-16T13:03:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_50': 2024-09-16T13:03:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:03:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:03:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14e4a0f2b037 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14e4a0f2b037 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:03:09 2024 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:03:09 2024 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:03:09 2024 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 26bf8c6e-9262-4fd0-975d-7cfaad8dde35 2024-09-16T13:03:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:03:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-16T13:03:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-16T13:03:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:03:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:03:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T13:03:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:03:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:03:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 25902080 bytes 2024-09-16T13:03:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 90546176 bytes 2024-09-16T13:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 149178368 bytes 2024-09-16T13:03:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 236121088 bytes 2024-09-16T13:03:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 282642432 bytes 2024-09-16T13:03:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 306366464 bytes 2024-09-16T13:03:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 312644608 bytes 2024-09-16T13:03:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 330507264 bytes 2024-09-16T13:03:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 356929536 bytes 2024-09-16T13:03:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 370196480 bytes 2024-09-16T13:03:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 401596416 bytes 2024-09-16T13:03:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 439856128 bytes 2024-09-16T13:03:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 484193280 bytes 2024-09-16T13:03:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 512994304 bytes 2024-09-16T13:03:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 547079168 bytes 2024-09-16T13:03:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 559071232 bytes 2024-09-16T13:03:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 575851520 bytes 2024-09-16T13:03:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 606512128 bytes 2024-09-16T13:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 616908800 bytes 2024-09-16T13:03:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 623037440 bytes 2024-09-16T13:03:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 661328896 bytes 2024-09-16T13:03:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 674988032 bytes 2024-09-16T13:03:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 715773952 bytes 2024-09-16T13:03:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 787637248 bytes 2024-09-16T13:03:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 788276224 bytes 2024-09-16T13:03:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 823055360 bytes 2024-09-16T13:03:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 895678464 bytes 2024-09-16T13:03:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 930118656 bytes 2024-09-16T13:03:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 943219712 bytes 2024-09-16T13:03:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 976299008 bytes 2024-09-16T13:03:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1001256960 bytes 2024-09-16T13:03:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1070829568 bytes 2024-09-16T13:03:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1097695232 bytes 2024-09-16T13:03:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1125326848 bytes 2024-09-16T13:03:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1153934336 bytes 2024-09-16T13:03:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1186304000 bytes 2024-09-16T13:03:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1211582464 bytes 2024-09-16T13:03:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1307879424 bytes 2024-09-16T13:03:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1328375808 bytes 2024-09-16T13:03:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1367523328 bytes 2024-09-16T13:03:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1433193472 bytes 2024-09-16T13:03:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1460998144 bytes 2024-09-16T13:03:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1483709440 bytes 2024-09-16T13:03:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1494155264 bytes 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=50 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-16T13:03:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-16T13:03:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:03:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:03:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:03:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T13:03:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-16T13:03:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-16T13:03:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=50 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 50 = 50 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-16T13:03:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:03:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:03:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:03:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T13:03:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-16T13:03:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-16T13:03:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=50 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:03:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:03:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:03:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:03:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-16T13:03:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-16T13:03:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=50 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 50 = 50 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-16T13:03:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:03:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:03:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:03:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:03:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-16T13:03:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-16T13:03:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=WARNING 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test WARNING = ERROR 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test WARNING = OK 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 2 -ne 1 2024-09-16T13:03:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 30 2024-09-16T13:03:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:03:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-16T13:03:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-16T13:03:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:03:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-16T13:03:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-16T13:03:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-16T13:03:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: compare many images' 2024-09-16T13:03:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T13:03:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T13:03:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 13:03:47 TEST: compare many images 2024-09-16T13:03:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 13:03:47 TEST: compare many images 2024-09-16T13:03:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:03:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:03:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:03:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:03:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-16T13:03:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:03:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:03:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:03:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:03:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:03:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:03:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:03:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:03:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:03:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:03:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:03:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-16T13:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 2 closest matches: 2024-09-16T13:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_10 2024-09-16T13:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_16 2024-09-16T13:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-16T13:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:03:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:03:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:03:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 6 closest matches: 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_11 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_13 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_15 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_17 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_18 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_19 2024-09-16T13:03:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:03:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:03:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-16T13:03:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:03:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:03:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:03:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:03:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:03:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:03:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:03:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-16T13:03:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 1 closest matches: 2024-09-16T13:03:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_12 2024-09-16T13:03:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:03:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:03:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:03:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:03:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:03:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:03:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:03:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:03:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:03:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:03:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-09-16T13:03:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:03:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:03:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:03:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:03:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_1.status 2024-09-16T13:03:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:03:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.status 2024-09-16T13:03:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-16T13:03:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_1.status 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-16T13:03:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 119 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.status 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.status 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-16T13:03:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:03:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=119 2024-09-16T13:03:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:03:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T13:03:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:03:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=68 2024-09-16T13:03:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 119 = 68 2024-09-16T13:03:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:03:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:03:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T13:03:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:03:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:03:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:03:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:03:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 119 = '' 2024-09-16T13:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:03:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T13:03:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:03:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:03:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:03:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:03:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:03:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 119 = '' 2024-09-16T13:03:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:03:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.status 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T13:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=119 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 119 = 119 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:03:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:03:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:03:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c962bf39ae 2024-09-16T13:03:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c962bf39ae 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:34 2024 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:34 2024 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:34 2024 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 70a8076e-7461-4223-9205-c586ea82527e 2024-09-16T13:03:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.export 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.export 2024-09-16T13:03:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.export 2024-09-16T13:03:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.export 2024-09-16T13:03:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:03:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.export 2024-09-16T13:03:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:03:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.export 2024-09-16T13:03:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_1.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_1.export 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:03:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:03:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:03:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:03:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-16T13:03:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:03:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:03:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:03:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:03:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:03:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:03:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:03:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-16T13:03:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:03:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:03:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:03:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:03:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:03:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:03:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:03:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:03:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-16T13:03:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:03:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:03:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:03:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:03:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-16T13:03:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_2.status 2024-09-16T13:03:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.status 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_2.status 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-16T13:03:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 121 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.status 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.status 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-16T13:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:03:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:03:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=121 2024-09-16T13:03:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:03:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:03:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T13:04:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:00.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=69 2024-09-16T13:04:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 121 = 69 2024-09-16T13:04:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T13:04:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:00.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 121 = '' 2024-09-16T13:04:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T13:04:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 121 = '' 2024-09-16T13:04:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.status 2024-09-16T13:04:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T13:04:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=121 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 121 = 121 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10cebd80bc8a 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10cebd80bc8a 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:41 2024 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:41 2024 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:41 2024 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5d32e8f9-d6e5-4ee3-9c1a-b169f80cfbb0 2024-09-16T13:04:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-16T13:04:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.export 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.export 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.export 2024-09-16T13:04:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.export 2024-09-16T13:04:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.export 2024-09-16T13:04:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.export 2024-09-16T13:04:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_2.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_2.export 2024-09-16T13:04:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-16T13:04:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-16T13:04:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-16T13:04:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 5 closest matches: 2024-09-16T13:04:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_31 2024-09-16T13:04:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_34 2024-09-16T13:04:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_35 2024-09-16T13:04:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_38 2024-09-16T13:04:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_39 2024-09-16T13:04:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-16T13:04:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-16T13:04:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-16T13:04:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_3.status 2024-09-16T13:04:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.status 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_3.status 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-16T13:04:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 123 2024-09-16T13:04:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.status 2024-09-16T13:04:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.status 2024-09-16T13:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-16T13:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=123 2024-09-16T13:04:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T13:04:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=70 2024-09-16T13:04:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 123 = 70 2024-09-16T13:04:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T13:04:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 123 = '' 2024-09-16T13:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T13:04:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 123 = '' 2024-09-16T13:04:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.status 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T13:04:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=123 2024-09-16T13:04:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 123 = 123 2024-09-16T13:04:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-16T13:04:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-16T13:04:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140734c1175 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140734c1175 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:54:48 2024 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:54:48 2024 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:54:48 2024 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: fa91e86c-1fbe-49d8-980f-ba4716f8e375 2024-09-16T13:04:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.export 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.export 2024-09-16T13:04:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.export 2024-09-16T13:04:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.export 2024-09-16T13:04:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.export 2024-09-16T13:04:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.export 2024-09-16T13:04:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_3.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_3.export 2024-09-16T13:04:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-16T13:04:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-16T13:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-16T13:04:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-16T13:04:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 2 closest matches: 2024-09-16T13:04:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_41 2024-09-16T13:04:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_46 2024-09-16T13:04:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-16T13:04:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 3 closest matches: 2024-09-16T13:04:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_42 2024-09-16T13:04:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_48 2024-09-16T13:04:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_49 2024-09-16T13:04:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:04:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:04:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-16T13:04:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 3 closest matches: 2024-09-16T13:04:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_40 2024-09-16T13:04:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_43 2024-09-16T13:04:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_47 2024-09-16T13:04:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2024-09-16T13:04:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:04:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:04:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:04:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2024-09-16T13:04:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-16T13:04:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_4.status 2024-09-16T13:04:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.status 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_4.status 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-16T13:04:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 125 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.status 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.status 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-16T13:04:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=125 2024-09-16T13:04:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T13:04:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=71 2024-09-16T13:04:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 125 = 71 2024-09-16T13:04:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T13:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 125 = '' 2024-09-16T13:04:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T13:04:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 125 = '' 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.status 2024-09-16T13:04:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T13:04:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=125 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 125 = 125 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1224492b43d1 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1224492b43d1 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:02 2024 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:02 2024 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:02 2024 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 9f626ce6-5b6e-44bf-be4c-04da712913b0 2024-09-16T13:04:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.export 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.export 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.export 2024-09-16T13:04:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.export 2024-09-16T13:04:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.export 2024-09-16T13:04:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.export 2024-09-16T13:04:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_4.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_4.export 2024-09-16T13:04:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-16T13:04:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-16T13:04:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-16T13:04:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:no valid command found; 1 closest matches: 2024-09-16T13:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status mirror/image_50 2024-09-16T13:04:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-16T13:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_5.status 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.status 2024-09-16T13:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-16T13:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_5.status 2024-09-16T13:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-16T13:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-16T13:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 127 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.status 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.status 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-16T13:04:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=127 2024-09-16T13:04:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T13:04:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=72 2024-09-16T13:04:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 127 = 72 2024-09-16T13:04:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T13:04:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 127 = '' 2024-09-16T13:04:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T13:04:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 127 = '' 2024-09-16T13:04:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.status 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T13:04:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=127 2024-09-16T13:04:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 127 = 127 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c929a4214e 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c929a4214e 2024-09-16T13:04:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:13 2024 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:13 2024 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:13 2024 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 435d688a-930a-49fa-824b-5ef9566d2d8e 2024-09-16T13:04:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.export 2024-09-16T13:04:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.export 2024-09-16T13:04:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.export 2024-09-16T13:04:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.export 2024-09-16T13:04:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.export 2024-09-16T13:04:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.export 2024-09-16T13:04:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_5.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_5.export 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:29.869 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T13:04:29.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T13:04:29.870 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T13:04:29.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:04:29.868+0000 7fdc5f5f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41007) UID: 1000 2024-09-16T13:04:30.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-16T13:04:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-16T13:04:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-16T13:04:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-16T13:04:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:04:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:04:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-16T13:04:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-16T13:04:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2024-09-16T13:04:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:04:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:04:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:04:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:04:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2024-09-16T13:04:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-16T13:04:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_6.status 2024-09-16T13:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.status 2024-09-16T13:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-16T13:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_6.status 2024-09-16T13:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-16T13:04:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-16T13:04:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 129 2024-09-16T13:04:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.status 2024-09-16T13:04:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.status 2024-09-16T13:04:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-16T13:04:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=129 2024-09-16T13:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T13:04:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=73 2024-09-16T13:04:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 129 = 73 2024-09-16T13:04:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T13:04:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 129 = '' 2024-09-16T13:04:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T13:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 129 = '' 2024-09-16T13:04:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:35.807 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T13:04:35.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T13:04:35.807 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T13:04:35.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:04:35.805+0000 7f82eb1f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 45463) UID: 1000 2024-09-16T13:04:36.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.status 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T13:04:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=129 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 129 = 129 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-16T13:04:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10cec47ab604 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10cec47ab604 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:20 2024 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:20 2024 2024-09-16T13:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:20 2024 2024-09-16T13:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 44d2ade4-d0ac-4f1d-a249-9712f1c710b8 2024-09-16T13:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.export 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.export 2024-09-16T13:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.export 2024-09-16T13:04:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.export 2024-09-16T13:04:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.export 2024-09-16T13:04:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.export 2024-09-16T13:04:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_6.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_6.export 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:37.871 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T13:04:37.871 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:04:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:04:37.871 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T13:04:37.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T13:04:37.872 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T13:04:37.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T13:04:37.875 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-16T13:04:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-16T13:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T13:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-16T13:04:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T13:04:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-16T13:04:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:04:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:04:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:04:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:04:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:04:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-16T13:04:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T13:04:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:04:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:04:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:04:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:04:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:04:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-09-16T13:04:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_7.status 2024-09-16T13:04:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.status 2024-09-16T13:04:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_7.status 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-16T13:04:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 131 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.status 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.status 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-16T13:04:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=131 2024-09-16T13:04:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T13:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 131 = '' 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T13:04:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 131 = '' 2024-09-16T13:04:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.status 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T13:04:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=131 2024-09-16T13:04:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 131 = 131 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:42.807 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T13:04:42.808 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:04:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:04:42.808 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T13:04:42.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T13:04:42.808 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T13:04:42.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T13:04:42.811 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-16T13:04:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-16T13:04:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-16T13:04:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140df06c1e 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140df06c1e 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:31 2024 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:31 2024 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:31 2024 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: e7069a5d-0e13-4d1b-976c-04e9409a6d09 2024-09-16T13:04:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-16T13:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.export 2024-09-16T13:04:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.export 2024-09-16T13:04:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.export 2024-09-16T13:04:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.export 2024-09-16T13:04:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.export 2024-09-16T13:04:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.export 2024-09-16T13:04:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_7.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_7.export 2024-09-16T13:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-16T13:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-16T13:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-16T13:04:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-16T13:04:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-16T13:04:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-16T13:04:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-16T13:04:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-16T13:04:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:04:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:04:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:04:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:04:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-09-16T13:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-16T13:04:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_8.status 2024-09-16T13:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.status 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_8.status 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-16T13:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 133 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.status 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.status 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-16T13:04:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=133 2024-09-16T13:04:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T13:04:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=75 2024-09-16T13:04:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 133 = 75 2024-09-16T13:04:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T13:04:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 133 = '' 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T13:04:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 133 = '' 2024-09-16T13:04:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.status 2024-09-16T13:04:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T13:04:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=133 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 133 = 133 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:04:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1224982abf6c 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1224982abf6c 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:04:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:43 2024 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:43 2024 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:43 2024 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 470fe0b0-f9fc-45ce-91f1-79e5c3c5fc0e 2024-09-16T13:04:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.export 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.export 2024-09-16T13:04:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.export 2024-09-16T13:04:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.export 2024-09-16T13:04:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.export 2024-09-16T13:04:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:04:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.export 2024-09-16T13:04:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_8.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_8.export 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:04:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:04:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:04:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:04:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:04:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:04:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:04:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:04:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:04:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:04:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:04:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:04:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:04:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:04:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:04:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-16T13:04:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-16T13:04:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:04:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:04:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:04:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:04:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:04:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:04:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:04:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:04:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:04:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:04:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-16T13:04:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:04:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:04:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:04:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:04:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-16T13:04:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_9.status 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.status 2024-09-16T13:04:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-16T13:04:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_9.status 2024-09-16T13:04:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-16T13:04:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 135 2024-09-16T13:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.status 2024-09-16T13:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:04:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.status 2024-09-16T13:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-16T13:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=135 2024-09-16T13:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T13:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=76 2024-09-16T13:04:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 135 = 76 2024-09-16T13:04:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T13:04:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 135 = '' 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:04:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:04:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:04:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:04:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:04:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T13:04:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:04:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:04:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:04:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:04:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:04:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:04:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:04:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 135 = '' 2024-09-16T13:04:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:04:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.status 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T13:05:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=135 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 135 = 135 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:05:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-16T13:05:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c9c6b05bf7 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c9c6b05bf7 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:55:55 2024 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:55:55 2024 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:55:55 2024 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: aa2d39b0-ecda-4092-b3c6-43b942830859 2024-09-16T13:05:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.export 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.export 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.export 2024-09-16T13:05:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.export 2024-09-16T13:05:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.export 2024-09-16T13:05:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.export 2024-09-16T13:05:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_9.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_9.export 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-16T13:05:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_10.status 2024-09-16T13:05:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.status 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_10.status 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-16T13:05:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 137 2024-09-16T13:05:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.status 2024-09-16T13:05:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.status 2024-09-16T13:05:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-16T13:05:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=137 2024-09-16T13:05:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T13:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=77 2024-09-16T13:05:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 137 = 77 2024-09-16T13:05:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T13:05:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=77 2024-09-16T13:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 137 = 77 2024-09-16T13:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T13:05:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 137 = '' 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.status 2024-09-16T13:05:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T13:05:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=137 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 137 = 137 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10cec745a29e 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10cec745a29e 2024-09-16T13:05:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:07 2024 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:07 2024 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:07 2024 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a9b282ed-f577-495d-bb2c-a0ac91558148 2024-09-16T13:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.export 2024-09-16T13:05:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.export 2024-09-16T13:05:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.export 2024-09-16T13:05:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.export 2024-09-16T13:05:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.export 2024-09-16T13:05:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.export 2024-09-16T13:05:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_10.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_10.export 2024-09-16T13:05:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-16T13:05:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-16T13:05:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:05:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:05:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:05:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-16T13:05:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_11.status 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.status 2024-09-16T13:05:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-16T13:05:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_11.status 2024-09-16T13:05:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-16T13:05:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-16T13:05:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 139 2024-09-16T13:05:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.status 2024-09-16T13:05:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.status 2024-09-16T13:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-16T13:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=139 2024-09-16T13:05:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T13:05:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=78 2024-09-16T13:05:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 139 = 78 2024-09-16T13:05:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T13:05:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 139 = '' 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T13:05:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 139 = '' 2024-09-16T13:05:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.status 2024-09-16T13:05:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T13:05:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=139 2024-09-16T13:05:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 139 = 139 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_11': 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 11404ab390d3 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.11404ab390d3 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:16 2024 2024-09-16T13:05:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:16 2024 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:16 2024 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 000b0e94-a5bd-4706-9afd-99ff1b8bf462 2024-09-16T13:05:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.export 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.export 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.export 2024-09-16T13:05:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.export 2024-09-16T13:05:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.export 2024-09-16T13:05:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.export 2024-09-16T13:05:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_11.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_11.export 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-16T13:05:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-16T13:05:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_12.status 2024-09-16T13:05:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.status 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_12.status 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-16T13:05:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 141 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.status 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.status 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-16T13:05:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=141 2024-09-16T13:05:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T13:05:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=79 2024-09-16T13:05:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 141 = 79 2024-09-16T13:05:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T13:05:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=79 2024-09-16T13:05:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 141 = 79 2024-09-16T13:05:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T13:05:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 141 = '' 2024-09-16T13:05:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.status 2024-09-16T13:05:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T13:05:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=141 2024-09-16T13:05:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 141 = 141 2024-09-16T13:05:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-16T13:05:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_12': 2024-09-16T13:05:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1224c0435b13 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1224c0435b13 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:30 2024 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:30 2024 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:30 2024 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 42c7f0d4-ea22-4303-82c7-5d306ec75295 2024-09-16T13:05:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.export 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.export 2024-09-16T13:05:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.export 2024-09-16T13:05:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.export 2024-09-16T13:05:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.export 2024-09-16T13:05:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.export 2024-09-16T13:05:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_12.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_12.export 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-16T13:05:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-16T13:05:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-16T13:05:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:05:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:05:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-16T13:05:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_13.status 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.status 2024-09-16T13:05:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-16T13:05:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_13.status 2024-09-16T13:05:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-16T13:05:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 143 2024-09-16T13:05:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.status 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.status 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-16T13:05:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=143 2024-09-16T13:05:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T13:05:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=80 2024-09-16T13:05:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 143 = 80 2024-09-16T13:05:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T13:05:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=80 2024-09-16T13:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 143 = 80 2024-09-16T13:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T13:05:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 143 = '' 2024-09-16T13:05:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.status 2024-09-16T13:05:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T13:05:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=143 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 143 = 143 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_13': 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c9a97f6a72 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c9a97f6a72 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:37 2024 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:37 2024 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:37 2024 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 451fa1a3-9713-4bdb-894c-abccb64adbc2 2024-09-16T13:05:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.export 2024-09-16T13:05:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.export 2024-09-16T13:05:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.export 2024-09-16T13:05:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.export 2024-09-16T13:05:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.export 2024-09-16T13:05:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.export 2024-09-16T13:05:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_13.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_13.export 2024-09-16T13:05:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-16T13:05:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-16T13:05:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-16T13:05:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-16T13:05:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:05:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:05:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:05:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-16T13:05:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-16T13:05:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-16T13:05:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:05:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:05:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:05:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:05:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-16T13:05:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-16T13:05:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:05:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-09-16T13:05:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-16T13:05:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_14.status 2024-09-16T13:05:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.status 2024-09-16T13:05:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-16T13:05:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_14.status 2024-09-16T13:05:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-16T13:05:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-16T13:05:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 145 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.status 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.status 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-16T13:05:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=145 2024-09-16T13:05:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T13:05:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=81 2024-09-16T13:05:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 145 = 81 2024-09-16T13:05:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T13:05:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=81 2024-09-16T13:05:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 145 = 81 2024-09-16T13:05:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T13:05:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 145 = '' 2024-09-16T13:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.status 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T13:05:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=145 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 145 = 145 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_14': 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ce8b37037f 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ce8b37037f 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:44 2024 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:44 2024 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:44 2024 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f3868c68-fe40-4444-8720-528d92e98de6 2024-09-16T13:05:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.export 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.export 2024-09-16T13:05:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.export 2024-09-16T13:05:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.export 2024-09-16T13:05:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.export 2024-09-16T13:05:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.export 2024-09-16T13:05:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_14.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_14.export 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-16T13:05:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-16T13:05:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-16T13:05:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-16T13:05:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:05:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:05:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:05:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-16T13:05:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_15.status 2024-09-16T13:05:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.status 2024-09-16T13:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-16T13:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_15.status 2024-09-16T13:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-16T13:05:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 147 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.status 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.status 2024-09-16T13:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-16T13:05:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=147 2024-09-16T13:05:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T13:05:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=82 2024-09-16T13:05:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 147 = 82 2024-09-16T13:05:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T13:05:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 147 = '' 2024-09-16T13:05:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:47.812 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5603303175460931) > weight (0.25) 2024-09-16T13:05:47.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.25334687632131125) > weight (0.25) 2024-09-16T13:05:47.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7600159381642446) > weight (0.25) 2024-09-16T13:05:47.813 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5701882311350559) > weight (0.25) 2024-09-16T13:05:47.813 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T13:05:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 147 = '' 2024-09-16T13:05:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:49.116 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.109+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dd205ba0 2024-09-16T13:05:49.117 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.112+0000 7fb62d3cd640 -1 reset not still connected to 0x5564de954820 2024-09-16T13:05:49.117 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.112+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e2310000 2024-09-16T13:05:49.117 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.112+0000 7fb62d3cd640 -1 reset not still connected to 0x5564dd205ba0 2024-09-16T13:05:49.117 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.112+0000 7fb62d3cd640 -1 reset not still connected to 0x5564de954820 2024-09-16T13:05:49.117 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:49.112+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e2310000 2024-09-16T13:05:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.status 2024-09-16T13:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T13:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=147 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 147 = 147 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-16T13:05:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_15': 2024-09-16T13:05:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 114031b6719d 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.114031b6719d 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:56:51 2024 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:56:51 2024 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:56:51 2024 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f65d50c0-9e44-4158-8c3d-91b35d21fe9a 2024-09-16T13:05:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.export 2024-09-16T13:05:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.export 2024-09-16T13:05:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.export 2024-09-16T13:05:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.export 2024-09-16T13:05:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.export 2024-09-16T13:05:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.export 2024-09-16T13:05:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_15.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_15.export 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:05:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-16T13:05:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:05:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:05:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:05:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-16T13:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:05:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:05:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:05:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:05:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-16T13:05:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-16T13:05:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-16T13:05:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:05:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:05:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:05:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:05:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-16T13:05:54.068 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:05:54.065+0000 7fb62d3cd640 -1 reset not still connected to 0x5564f800a4e0 2024-09-16T13:05:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-16T13:05:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:05:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:05:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:05:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:05:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-16T13:05:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-16T13:05:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:05:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:05:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:05:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:05:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:05:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:05:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-16T13:05:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:05:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:05:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:05:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:05:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:05:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:05:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:05:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-16T13:05:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_16.status 2024-09-16T13:05:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.status 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_16.status 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-16T13:05:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 149 2024-09-16T13:05:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.status 2024-09-16T13:05:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.status 2024-09-16T13:05:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-16T13:05:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:05:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=149 2024-09-16T13:05:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T13:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=83 2024-09-16T13:05:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 149 = 83 2024-09-16T13:05:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T13:05:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 149 = '' 2024-09-16T13:05:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:05:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T13:05:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:05:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:05:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:05:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:05:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:05:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 149 = '' 2024-09-16T13:05:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:05:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.status 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T13:05:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:05:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=149 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 149 = 149 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:05:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:05:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:05:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_16': 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1224d476821 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1224d476821 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T13:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f77bbb21-fc88-4735-8cf0-73c5d5cc573f 2024-09-16T13:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.export 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.export 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.export 2024-09-16T13:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.export 2024-09-16T13:05:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.export 2024-09-16T13:05:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:05:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.export 2024-09-16T13:05:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_16.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_16.export 2024-09-16T13:05:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:05:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-16T13:06:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-16T13:06:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-16T13:06:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-16T13:06:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-16T13:06:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:02.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_17.status 2024-09-16T13:06:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.status 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_17.status 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-16T13:06:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 151 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.status 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.status 2024-09-16T13:06:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-16T13:06:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=151 2024-09-16T13:06:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T13:06:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=84 2024-09-16T13:06:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 151 = 84 2024-09-16T13:06:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T13:06:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 151 = '' 2024-09-16T13:06:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T13:06:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 151 = '' 2024-09-16T13:06:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.status 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T13:06:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=151 2024-09-16T13:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 151 = 151 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:06.875 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T13:06:06.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T13:06:06.876 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T13:06:06.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:06:06.875+0000 7f93831f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53760) UID: 1000 2024-09-16T13:06:07.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:06:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_17': 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c935f21bed 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c935f21bed 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:16 2024 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:16 2024 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:16 2024 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 105a819b-f484-4d1d-a16e-4f48810acd47 2024-09-16T13:06:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.export 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.export 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.export 2024-09-16T13:06:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.export 2024-09-16T13:06:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.export 2024-09-16T13:06:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.export 2024-09-16T13:06:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_17.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_17.export 2024-09-16T13:06:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:07.876 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T13:06:07.877 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:06:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:06:07.877 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T13:06:07.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T13:06:07.877 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T13:06:07.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T13:06:07.880 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-16T13:06:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-16T13:06:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-16T13:06:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-16T13:06:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-16T13:06:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-16T13:06:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_18.status 2024-09-16T13:06:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.status 2024-09-16T13:06:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-16T13:06:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_18.status 2024-09-16T13:06:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-16T13:06:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 153 2024-09-16T13:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.status 2024-09-16T13:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.status 2024-09-16T13:06:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-16T13:06:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=153 2024-09-16T13:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T13:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=85 2024-09-16T13:06:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 153 = 85 2024-09-16T13:06:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T13:06:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 153 = '' 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T13:06:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 153 = '' 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.status 2024-09-16T13:06:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T13:06:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=153 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 153 = 153 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_18': 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ce167fc664 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ce167fc664 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:32 2024 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:32 2024 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:32 2024 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d03e92a5-c8e6-4505-aae2-e8c01ac781fc 2024-09-16T13:06:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.export 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.export 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.export 2024-09-16T13:06:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.export 2024-09-16T13:06:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.export 2024-09-16T13:06:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.export 2024-09-16T13:06:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_18.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_18.export 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-16T13:06:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-16T13:06:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-09-16T13:06:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-16T13:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_19.status 2024-09-16T13:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.status 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_19.status 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-16T13:06:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 155 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.status 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.status 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-16T13:06:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=155 2024-09-16T13:06:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T13:06:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=86 2024-09-16T13:06:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 155 = 86 2024-09-16T13:06:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T13:06:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 155 = '' 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T13:06:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 155 = '' 2024-09-16T13:06:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.status 2024-09-16T13:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T13:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=155 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 155 = 155 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-16T13:06:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_19': 2024-09-16T13:06:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140215832d2 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140215832d2 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:43 2024 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:43 2024 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:43 2024 2024-09-16T13:06:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7c010caf-1b79-4274-b3dd-76aca299a0a4 2024-09-16T13:06:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.export 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.export 2024-09-16T13:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.export 2024-09-16T13:06:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.export 2024-09-16T13:06:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.export 2024-09-16T13:06:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.export 2024-09-16T13:06:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_19.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_19.export 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-16T13:06:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-16T13:06:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-16T13:06:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-16T13:06:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-16T13:06:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-16T13:06:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:06:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:06:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-09-16T13:06:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_20.status 2024-09-16T13:06:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.status 2024-09-16T13:06:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-16T13:06:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_20.status 2024-09-16T13:06:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-16T13:06:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 157 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.status 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.status 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-16T13:06:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=157 2024-09-16T13:06:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T13:06:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=87 2024-09-16T13:06:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 157 = 87 2024-09-16T13:06:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T13:06:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 157 = '' 2024-09-16T13:06:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T13:06:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 157 = '' 2024-09-16T13:06:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.status 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T13:06:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=157 2024-09-16T13:06:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 157 = 157 2024-09-16T13:06:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-16T13:06:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_20': 2024-09-16T13:06:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12248ebd36ab 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12248ebd36ab 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:57:58 2024 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:57:58 2024 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:57:58 2024 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 897043d8-7460-4263-8c01-2902509580ba 2024-09-16T13:06:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-16T13:06:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:06:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.export 2024-09-16T13:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.export 2024-09-16T13:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.export 2024-09-16T13:06:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.export 2024-09-16T13:06:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.export 2024-09-16T13:06:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.export 2024-09-16T13:06:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_20.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_20.export 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-16T13:06:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-16T13:06:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-16T13:06:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_21.status 2024-09-16T13:06:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.status 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_21.status 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-16T13:06:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 159 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.status 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.status 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-16T13:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=159 2024-09-16T13:06:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T13:06:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=88 2024-09-16T13:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 159 = 88 2024-09-16T13:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T13:06:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 159 = '' 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T13:06:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 159 = '' 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.status 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T13:06:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=159 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 159 = 159 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_21': 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129b6d62f0b7 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129b6d62f0b7 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:10 2024 2024-09-16T13:06:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:10 2024 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:10 2024 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5ec8e0c7-23d8-4f56-a4da-5c7e8d8a4880 2024-09-16T13:06:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.export 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.export 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.export 2024-09-16T13:06:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.export 2024-09-16T13:06:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.export 2024-09-16T13:06:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.export 2024-09-16T13:06:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_21.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_21.export 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-16T13:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-16T13:06:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-16T13:06:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-16T13:06:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-16T13:06:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:06:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:06:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:06:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-16T13:06:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_22.status 2024-09-16T13:06:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.status 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_22.status 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-16T13:06:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 161 2024-09-16T13:06:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.status 2024-09-16T13:06:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.status 2024-09-16T13:06:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-16T13:06:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=161 2024-09-16T13:06:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T13:06:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=89 2024-09-16T13:06:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 161 = 89 2024-09-16T13:06:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T13:06:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 161 = '' 2024-09-16T13:06:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T13:06:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 161 = '' 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.status 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T13:06:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=161 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 161 = 161 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_22': 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129bec9ecc18 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129bec9ecc18 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:17 2024 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:17 2024 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:17 2024 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 4135e1aa-5a4e-4f22-a4a4-d979c6386191 2024-09-16T13:06:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.export 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.export 2024-09-16T13:06:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.export 2024-09-16T13:06:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.export 2024-09-16T13:06:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.export 2024-09-16T13:06:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.export 2024-09-16T13:06:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_22.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_22.export 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:06:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:06:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:06:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:06:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:06:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:06:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-16T13:06:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-16T13:06:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:06:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:06:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:06:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-16T13:06:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-16T13:06:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:06:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:06:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:06:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-16T13:06:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-16T13:06:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-16T13:06:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_23.status 2024-09-16T13:06:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.status 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_23.status 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-16T13:06:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 163 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.status 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.status 2024-09-16T13:06:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-16T13:06:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:06:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=163 2024-09-16T13:06:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T13:06:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=90 2024-09-16T13:06:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 163 = 90 2024-09-16T13:06:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T13:06:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 163 = '' 2024-09-16T13:06:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:06:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T13:06:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:06:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 163 = '' 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:06:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:06:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.status 2024-09-16T13:06:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T13:06:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:06:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=163 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 163 = 163 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:06:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_23': 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140524e7364 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140524e7364 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:32 2024 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:32 2024 2024-09-16T13:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:32 2024 2024-09-16T13:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: deee2aeb-a92c-4e22-ab10-fb00148a0183 2024-09-16T13:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:06:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.export 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.export 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.export 2024-09-16T13:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.export 2024-09-16T13:06:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.export 2024-09-16T13:06:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:06:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.export 2024-09-16T13:06:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_23.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_23.export 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:06:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:06:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:06:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-16T13:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-16T13:07:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-16T13:07:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-16T13:07:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-16T13:07:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-16T13:07:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-16T13:07:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-16T13:07:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-16T13:07:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-16T13:07:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_24.status 2024-09-16T13:07:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.status 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_24.status 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-16T13:07:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-16T13:07:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 165 2024-09-16T13:07:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.status 2024-09-16T13:07:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.status 2024-09-16T13:07:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-16T13:07:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=165 2024-09-16T13:07:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T13:07:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=91 2024-09-16T13:07:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 165 = 91 2024-09-16T13:07:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T13:07:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 165 = '' 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T13:07:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 165 = '' 2024-09-16T13:07:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.status 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T13:07:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=165 2024-09-16T13:07:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 165 = 165 2024-09-16T13:07:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-16T13:07:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-16T13:07:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_24': 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129be0cf6c9f 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129be0cf6c9f 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:46 2024 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:46 2024 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:46 2024 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d84f0610-c1af-4671-8b4a-aeb184619555 2024-09-16T13:07:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-16T13:07:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.export 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.export 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.export 2024-09-16T13:07:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.export 2024-09-16T13:07:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.export 2024-09-16T13:07:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.export 2024-09-16T13:07:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_24.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_24.export 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:07.881 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T13:07:07.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T13:07:07.882 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T13:07:07.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:07:07.880+0000 7f257c3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 56671) UID: 1000 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-16T13:07:08.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:07:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-16T13:07:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-16T13:07:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-16T13:07:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-16T13:07:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:07:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:07:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:07:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-09-16T13:07:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_25.status 2024-09-16T13:07:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.status 2024-09-16T13:07:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-16T13:07:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_25.status 2024-09-16T13:07:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-16T13:07:08.883 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T13:07:08.883 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:07:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:07:08.883 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T13:07:08.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T13:07:08.884 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T13:07:08.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T13:07:08.887 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-16T13:07:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 167 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.status 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.status 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-16T13:07:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=167 2024-09-16T13:07:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T13:07:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=92 2024-09-16T13:07:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 167 = 92 2024-09-16T13:07:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T13:07:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 167 = '' 2024-09-16T13:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T13:07:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 167 = '' 2024-09-16T13:07:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.status 2024-09-16T13:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T13:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=167 2024-09-16T13:07:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 167 = 167 2024-09-16T13:07:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-16T13:07:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_25': 2024-09-16T13:07:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c94a69dfc0 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c94a69dfc0 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:58:59 2024 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:58:59 2024 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:58:59 2024 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: dde89dcd-ca3e-4c90-a12d-78663eac03bf 2024-09-16T13:07:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.export 2024-09-16T13:07:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.export 2024-09-16T13:07:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.export 2024-09-16T13:07:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.export 2024-09-16T13:07:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.export 2024-09-16T13:07:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.export 2024-09-16T13:07:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_25.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_25.export 2024-09-16T13:07:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-16T13:07:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-16T13:07:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-16T13:07:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:16.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T13:07:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T13:07:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T13:07:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T13:07:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-16T13:07:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T13:07:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-16T13:07:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:07:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:07:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:07:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:07:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:07:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:07:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-16T13:07:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-16T13:07:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_26.status 2024-09-16T13:07:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.status 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_26.status 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-16T13:07:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 169 2024-09-16T13:07:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.status 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.status 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-16T13:07:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=169 2024-09-16T13:07:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T13:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=93 2024-09-16T13:07:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 169 = 93 2024-09-16T13:07:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T13:07:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 169 = '' 2024-09-16T13:07:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T13:07:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 169 = '' 2024-09-16T13:07:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.status 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T13:07:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=169 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 169 = 169 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_26': 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ce5a52c1e0 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ce5a52c1e0 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:05 2024 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:05 2024 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:05 2024 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3b3040ad-963e-4bf9-8b3f-18deb1f76778 2024-09-16T13:07:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.export 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.export 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.export 2024-09-16T13:07:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.export 2024-09-16T13:07:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.export 2024-09-16T13:07:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.export 2024-09-16T13:07:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_26.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_26.export 2024-09-16T13:07:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-16T13:07:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-16T13:07:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-16T13:07:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-16T13:07:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-16T13:07:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-16T13:07:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-16T13:07:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-16T13:07:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:07:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:07:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:07:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:07:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-16T13:07:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-16T13:07:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_27.status 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.status 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_27.status 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-16T13:07:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 171 2024-09-16T13:07:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.status 2024-09-16T13:07:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.status 2024-09-16T13:07:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-16T13:07:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=171 2024-09-16T13:07:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T13:07:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=94 2024-09-16T13:07:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 171 = 94 2024-09-16T13:07:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T13:07:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 171 = '' 2024-09-16T13:07:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T13:07:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 171 = '' 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.status 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T13:07:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=171 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 171 = 171 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-16T13:07:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_27': 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 114017482ded 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.114017482ded 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:13 2024 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:13 2024 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:13 2024 2024-09-16T13:07:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6e695d31-7fbb-4a6d-bf1c-da34f746931f 2024-09-16T13:07:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-16T13:07:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.export 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.export 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.export 2024-09-16T13:07:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.export 2024-09-16T13:07:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.export 2024-09-16T13:07:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.export 2024-09-16T13:07:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_27.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_27.export 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-16T13:07:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-16T13:07:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-16T13:07:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-16T13:07:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-16T13:07:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-16T13:07:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-16T13:07:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-16T13:07:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_28.status 2024-09-16T13:07:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.status 2024-09-16T13:07:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-16T13:07:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_28.status 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-16T13:07:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 173 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.status 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.status 2024-09-16T13:07:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-16T13:07:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=173 2024-09-16T13:07:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T13:07:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 173 = '' 2024-09-16T13:07:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T13:07:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 173 = '' 2024-09-16T13:07:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T13:07:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 173 = '' 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.status 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T13:07:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=173 2024-09-16T13:07:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 173 = 173 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_28': 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129bbf84fc71 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129bbf84fc71 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:22 2024 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:22 2024 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:22 2024 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ed51faf8-e2c3-44a4-9d09-787be9fd78f9 2024-09-16T13:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.export 2024-09-16T13:07:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.export 2024-09-16T13:07:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.export 2024-09-16T13:07:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.export 2024-09-16T13:07:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.export 2024-09-16T13:07:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.export 2024-09-16T13:07:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_28.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_28.export 2024-09-16T13:07:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-16T13:07:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:40.814 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T13:07:40.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T13:07:40.814 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T13:07:40.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:07:40.813+0000 7f80a09f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 54012) UID: 1000 2024-09-16T13:07:41.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-16T13:07:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-16T13:07:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-16T13:07:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_29.status 2024-09-16T13:07:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.status 2024-09-16T13:07:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-16T13:07:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_29.status 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-16T13:07:41.815 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T13:07:41.815 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:07:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:07:41.815 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T13:07:41.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T13:07:41.816 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T13:07:41.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T13:07:41.819 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-16T13:07:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 175 2024-09-16T13:07:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.status 2024-09-16T13:07:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.status 2024-09-16T13:07:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-16T13:07:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=175 2024-09-16T13:07:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T13:07:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=96 2024-09-16T13:07:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 175 = 96 2024-09-16T13:07:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T13:07:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 175 = '' 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T13:07:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 175 = '' 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.status 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=175 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 175 = 175 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_29': 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c98e5cf61f 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c98e5cf61f 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:35 2024 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:35 2024 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:35 2024 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 21593e92-6a4d-4963-8412-532dd99efda2 2024-09-16T13:07:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.export 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.export 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.export 2024-09-16T13:07:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.export 2024-09-16T13:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.export 2024-09-16T13:07:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.export 2024-09-16T13:07:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_29.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_29.export 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-16T13:07:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_30.status 2024-09-16T13:07:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.status 2024-09-16T13:07:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_30.status 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-16T13:07:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 177 2024-09-16T13:07:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.status 2024-09-16T13:07:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.status 2024-09-16T13:07:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-16T13:07:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=177 2024-09-16T13:07:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T13:07:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=97 2024-09-16T13:07:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 177 = 97 2024-09-16T13:07:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T13:07:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 177 = '' 2024-09-16T13:07:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T13:07:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 177 = '' 2024-09-16T13:07:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:07:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.status 2024-09-16T13:07:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T13:07:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=177 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 177 = 177 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:07:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_30': 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ced0bdecb5 2024-09-16T13:07:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ced0bdecb5 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:41 2024 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:41 2024 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:41 2024 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 795f020a-5b0a-43d5-a652-615b46e9624a 2024-09-16T13:07:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:07:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-16T13:07:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:07:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:07:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.export 2024-09-16T13:07:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.export 2024-09-16T13:07:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.export 2024-09-16T13:07:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.export 2024-09-16T13:07:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.export 2024-09-16T13:07:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:07:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.export 2024-09-16T13:07:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_30.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_30.export 2024-09-16T13:07:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:07:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:07:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-16T13:07:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:07:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:07:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:07:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-16T13:07:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:07:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:07:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:07:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:07:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:07:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:07:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:07:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-16T13:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-16T13:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:07:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:07:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:07:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-16T13:07:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-16T13:07:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:07:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:07:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:07:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:07:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:07:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:07:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-16T13:07:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:07:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:07:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:07:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:07:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-16T13:07:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_31.status 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.status 2024-09-16T13:07:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-16T13:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_31.status 2024-09-16T13:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-16T13:07:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-16T13:07:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 179 2024-09-16T13:07:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.status 2024-09-16T13:07:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:07:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.status 2024-09-16T13:07:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-16T13:07:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:07:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=179 2024-09-16T13:07:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:07:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T13:07:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=98 2024-09-16T13:07:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 179 = 98 2024-09-16T13:07:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:07:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T13:07:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 179 = '' 2024-09-16T13:07:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T13:07:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:07:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:07:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:07:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:07:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:07:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:07:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:07:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 179 = '' 2024-09-16T13:07:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:07:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:08:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.status 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T13:08:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=179 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 179 = 179 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_31': 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140b7567a3 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140b7567a3 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 12:59:55 2024 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 12:59:55 2024 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 12:59:55 2024 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: eeb2f380-5802-4457-936c-f678d743f57e 2024-09-16T13:08:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.export 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.export 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.export 2024-09-16T13:08:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.export 2024-09-16T13:08:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.export 2024-09-16T13:08:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.export 2024-09-16T13:08:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_31.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_31.export 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-16T13:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-16T13:08:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-16T13:08:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-16T13:08:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-16T13:08:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_32.status 2024-09-16T13:08:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.status 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_32.status 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-16T13:08:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 181 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.status 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.status 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-16T13:08:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=181 2024-09-16T13:08:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T13:08:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=99 2024-09-16T13:08:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 181 = 99 2024-09-16T13:08:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T13:08:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 181 = '' 2024-09-16T13:08:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T13:08:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 181 = '' 2024-09-16T13:08:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.status 2024-09-16T13:08:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T13:08:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=181 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 181 = 181 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-16T13:08:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_32': 2024-09-16T13:08:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129bd8465455 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129bd8465455 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:02 2024 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:02 2024 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:02 2024 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7bac516b-dc94-4b85-9cd0-8bcb96179e9a 2024-09-16T13:08:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.export 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.export 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.export 2024-09-16T13:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.export 2024-09-16T13:08:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.export 2024-09-16T13:08:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.export 2024-09-16T13:08:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_32.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_32.export 2024-09-16T13:08:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-16T13:08:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-16T13:08:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-16T13:08:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-16T13:08:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_33.status 2024-09-16T13:08:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.status 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_33.status 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-16T13:08:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 183 2024-09-16T13:08:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.status 2024-09-16T13:08:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.status 2024-09-16T13:08:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-16T13:08:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=183 2024-09-16T13:08:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T13:08:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 183 = '' 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T13:08:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 183 = '' 2024-09-16T13:08:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.status 2024-09-16T13:08:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T13:08:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=183 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 183 = 183 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_33': 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c9b795016b 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c9b795016b 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:14 2024 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:14 2024 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:14 2024 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 686e94e1-8f41-41d6-998f-9112a95df1d3 2024-09-16T13:08:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.export 2024-09-16T13:08:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.export 2024-09-16T13:08:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.export 2024-09-16T13:08:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.export 2024-09-16T13:08:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.export 2024-09-16T13:08:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.export 2024-09-16T13:08:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_33.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_33.export 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-16T13:08:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-16T13:08:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-16T13:08:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_34.status 2024-09-16T13:08:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.status 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_34.status 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-16T13:08:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-16T13:08:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 185 2024-09-16T13:08:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.status 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.status 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-16T13:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=185 2024-09-16T13:08:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T13:08:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=101 2024-09-16T13:08:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 185 = 101 2024-09-16T13:08:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T13:08:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 185 = '' 2024-09-16T13:08:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.status 2024-09-16T13:08:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T13:08:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=185 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 185 = 185 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-16T13:08:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_34': 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ce24d201dd 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ce24d201dd 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:21 2024 2024-09-16T13:08:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:21 2024 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:21 2024 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 321f4d24-c15c-4025-9b78-f2e2dd17716a 2024-09-16T13:08:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-16T13:08:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.export 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.export 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.export 2024-09-16T13:08:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.export 2024-09-16T13:08:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.export 2024-09-16T13:08:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.export 2024-09-16T13:08:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_34.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_34.export 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-16T13:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-16T13:08:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_35.status 2024-09-16T13:08:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.status 2024-09-16T13:08:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-16T13:08:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_35.status 2024-09-16T13:08:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-16T13:08:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-16T13:08:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 187 2024-09-16T13:08:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.status 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.status 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-16T13:08:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=187 2024-09-16T13:08:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T13:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=102 2024-09-16T13:08:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 187 = 102 2024-09-16T13:08:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T13:08:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 187 = '' 2024-09-16T13:08:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T13:08:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 187 = '' 2024-09-16T13:08:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.status 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T13:08:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=187 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 187 = 187 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_35': 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 114031b2b5b3 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.114031b2b5b3 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:30 2024 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:30 2024 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:30 2024 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1bac1448-c694-4f80-90b6-6bd1f7068529 2024-09-16T13:08:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.export 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.export 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.export 2024-09-16T13:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.export 2024-09-16T13:08:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.export 2024-09-16T13:08:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.export 2024-09-16T13:08:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_35.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_35.export 2024-09-16T13:08:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T13:08:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T13:08:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:08:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:08:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:08:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T13:08:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T13:08:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:08:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T13:08:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T13:08:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:08:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:08:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:08:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:08:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:08:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:08:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-16T13:08:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-16T13:08:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_36.status 2024-09-16T13:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.status 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_36.status 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-16T13:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 189 2024-09-16T13:08:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.status 2024-09-16T13:08:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.status 2024-09-16T13:08:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-16T13:08:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=189 2024-09-16T13:08:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T13:08:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=103 2024-09-16T13:08:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 189 = 103 2024-09-16T13:08:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T13:08:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 189 = '' 2024-09-16T13:08:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T13:08:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 189 = '' 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:08:37.888 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.30720821094394435) > weight (0.25) 2024-09-16T13:08:37.888 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T13:08:37.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T13:08:37.889 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T13:08:37.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:08:37.887+0000 7fb7095f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 51782) UID: 1000 2024-09-16T13:08:38.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:08:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.status 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T13:08:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=189 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 189 = 189 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-16T13:08:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_36': 2024-09-16T13:08:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129bfcb27d25 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129bfcb27d25 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:41 2024 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:41 2024 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:41 2024 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 573c8bbb-eb7d-47c7-a3f5-dd7fbc895516 2024-09-16T13:08:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.export 2024-09-16T13:08:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.export 2024-09-16T13:08:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.export 2024-09-16T13:08:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.export 2024-09-16T13:08:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.export 2024-09-16T13:08:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.export 2024-09-16T13:08:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_36.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_36.export 2024-09-16T13:08:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-16T13:08:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-16T13:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-16T13:08:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-16T13:08:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-16T13:08:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-16T13:08:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:08:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:08:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:08:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:08:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-16T13:08:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:08:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:08:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:08:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-16T13:08:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-16T13:08:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_37.status 2024-09-16T13:08:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.status 2024-09-16T13:08:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-16T13:08:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_37.status 2024-09-16T13:08:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-16T13:08:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 191 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.status 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.status 2024-09-16T13:08:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-16T13:08:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=191 2024-09-16T13:08:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T13:08:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 191 = '' 2024-09-16T13:08:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T13:08:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 191 = '' 2024-09-16T13:08:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.status 2024-09-16T13:08:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T13:08:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=191 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 191 = 191 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-16T13:08:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_37': 2024-09-16T13:08:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c9fb9bf1ba 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c9fb9bf1ba 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:00:48 2024 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:00:48 2024 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:00:48 2024 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: da41c0e4-7e5d-4612-8458-a799307ed96f 2024-09-16T13:08:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.export 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.export 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.export 2024-09-16T13:08:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.export 2024-09-16T13:08:46.890 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T13:08:46.890 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:08:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:08:46.891 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T13:08:46.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T13:08:46.891 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T13:08:46.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T13:08:46.894 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-16T13:08:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.export 2024-09-16T13:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.export 2024-09-16T13:08:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_37.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_37.export 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-16T13:08:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-16T13:08:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-16T13:08:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-16T13:08:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:08:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:08:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:08:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-16T13:08:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-16T13:08:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:08:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:08:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:08:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-16T13:08:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-16T13:08:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_38.status 2024-09-16T13:08:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.status 2024-09-16T13:08:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-16T13:08:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_38.status 2024-09-16T13:08:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-16T13:08:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-16T13:08:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 193 2024-09-16T13:08:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.status 2024-09-16T13:08:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.status 2024-09-16T13:08:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-16T13:08:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=193 2024-09-16T13:08:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T13:08:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 193 = '' 2024-09-16T13:08:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T13:08:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 193 = '' 2024-09-16T13:08:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.status 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T13:08:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=193 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 193 = 193 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_38': 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10ce74748ff1 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10ce74748ff1 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:08:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:01 2024 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:01 2024 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:01 2024 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 721e34a7-a4b2-4332-a181-ccded18e8696 2024-09-16T13:08:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.export 2024-09-16T13:08:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.export 2024-09-16T13:08:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.export 2024-09-16T13:08:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.export 2024-09-16T13:08:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.export 2024-09-16T13:08:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:08:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.export 2024-09-16T13:08:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_38.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_38.export 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:08:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-16T13:08:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:08:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:08:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:08:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:08:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:08:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:08:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:08:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-16T13:08:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-16T13:08:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:08:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:08:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:08:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:08:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:08:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-16T13:08:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-16T13:08:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:08:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:08:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:08:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-16T13:08:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:08:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:08:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:08:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:08:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-16T13:08:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:08:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:08:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:08:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:08:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:08:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:08:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:08:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-16T13:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_39.status 2024-09-16T13:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.status 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_39.status 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-16T13:08:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 195 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.status 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.status 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-16T13:08:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:08:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=195 2024-09-16T13:08:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T13:08:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 195 = '' 2024-09-16T13:08:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T13:08:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:08:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:08:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:08:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:08:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:08:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 195 = '' 2024-09-16T13:08:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:08:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:08:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.status 2024-09-16T13:08:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T13:08:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:08:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:08:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=195 2024-09-16T13:08:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 195 = 195 2024-09-16T13:08:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:08:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:08:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_39': 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 114032eae50e 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.114032eae50e 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:08 2024 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:08 2024 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:08 2024 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: fe577bd1-b7b2-46e6-adee-f694b7fb7801 2024-09-16T13:09:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.export 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.export 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.export 2024-09-16T13:09:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.export 2024-09-16T13:09:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.export 2024-09-16T13:09:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.export 2024-09-16T13:09:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_39.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_39.export 2024-09-16T13:09:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-16T13:09:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-16T13:09:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-16T13:09:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-16T13:09:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_40.status 2024-09-16T13:09:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.status 2024-09-16T13:09:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-16T13:09:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_40.status 2024-09-16T13:09:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-16T13:09:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 197 2024-09-16T13:09:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.status 2024-09-16T13:09:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.status 2024-09-16T13:09:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-16T13:09:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=197 2024-09-16T13:09:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T13:09:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 197 = '' 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T13:09:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 197 = '' 2024-09-16T13:09:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.status 2024-09-16T13:09:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T13:09:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=197 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 197 = 197 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-16T13:09:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_40': 2024-09-16T13:09:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129b918b5532 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129b918b5532 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:23 2024 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:23 2024 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:23 2024 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a44aec3e-055a-42d9-a75a-50b8b1713251 2024-09-16T13:09:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.export 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.export 2024-09-16T13:09:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.export 2024-09-16T13:09:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.export 2024-09-16T13:09:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.export 2024-09-16T13:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.export 2024-09-16T13:09:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_40.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_40.export 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-16T13:09:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-16T13:09:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T13:09:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-16T13:09:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T13:09:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:09:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:09:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:09:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-16T13:09:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T13:09:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:09:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:09:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:09:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-16T13:09:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-16T13:09:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_41.status 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.status 2024-09-16T13:09:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-16T13:09:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_41.status 2024-09-16T13:09:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-16T13:09:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 199 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.status 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.status 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-16T13:09:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=199 2024-09-16T13:09:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T13:09:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 199 = '' 2024-09-16T13:09:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T13:09:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 199 = '' 2024-09-16T13:09:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.status 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T13:09:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=199 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 199 = 199 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-16T13:09:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_41': 2024-09-16T13:09:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12b1f1436b0e 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12b1f1436b0e 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:32 2024 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:32 2024 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:32 2024 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3e6603df-4a85-45e2-92bc-3718de0d61a1 2024-09-16T13:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.export 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.export 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.export 2024-09-16T13:09:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.export 2024-09-16T13:09:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.export 2024-09-16T13:09:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.export 2024-09-16T13:09:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_41.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_41.export 2024-09-16T13:09:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-16T13:09:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-16T13:09:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-16T13:09:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-16T13:09:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_42.status 2024-09-16T13:09:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.status 2024-09-16T13:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-16T13:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_42.status 2024-09-16T13:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-16T13:09:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 201 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.status 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.status 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-16T13:09:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=201 2024-09-16T13:09:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T13:09:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 201 = '' 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T13:09:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 201 = '' 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.status 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T13:09:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=201 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 201 = 201 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_42': 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12b1d30e8c2c 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12b1d30e8c2c 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:38 2024 2024-09-16T13:09:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:38 2024 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:38 2024 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 362a7f1e-b75e-484c-bd35-6308692dbcfe 2024-09-16T13:09:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.export 2024-09-16T13:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.export 2024-09-16T13:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.export 2024-09-16T13:09:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.export 2024-09-16T13:09:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.export 2024-09-16T13:09:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.export 2024-09-16T13:09:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_42.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_42.export 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:22.819 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T13:09:22.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T13:09:22.820 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-16T13:09:22.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:09:22.819+0000 7f03775f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 59674) UID: 1000 2024-09-16T13:09:23.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:09:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-16T13:09:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:24.821 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T13:09:24.821 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:09:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:09:24.822 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T13:09:24.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T13:09:24.822 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T13:09:24.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T13:09:24.825 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-16T13:09:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-16T13:09:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-16T13:09:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-16T13:09:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-16T13:09:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_43.status 2024-09-16T13:09:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.status 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_43.status 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-16T13:09:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 203 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.status 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.status 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-16T13:09:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=203 2024-09-16T13:09:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T13:09:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 203 = '' 2024-09-16T13:09:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T13:09:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 203 = '' 2024-09-16T13:09:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.status 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T13:09:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=203 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 203 = 203 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_43': 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129bfb32cde7 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129bfb32cde7 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:01:51 2024 2024-09-16T13:09:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:01:51 2024 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:01:51 2024 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 89399335-667d-40ff-83af-2f8d80ec3800 2024-09-16T13:09:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-16T13:09:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.export 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.export 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.export 2024-09-16T13:09:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.export 2024-09-16T13:09:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.export 2024-09-16T13:09:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.export 2024-09-16T13:09:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_43.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_43.export 2024-09-16T13:09:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-16T13:09:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-16T13:09:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-16T13:09:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T13:09:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-16T13:09:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_44.status 2024-09-16T13:09:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.status 2024-09-16T13:09:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-16T13:09:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_44.status 2024-09-16T13:09:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-16T13:09:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 205 2024-09-16T13:09:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.status 2024-09-16T13:09:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.status 2024-09-16T13:09:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-16T13:09:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=205 2024-09-16T13:09:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T13:09:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 205 = '' 2024-09-16T13:09:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T13:09:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 205 = '' 2024-09-16T13:09:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.status 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T13:09:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=205 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 205 = 205 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_44': 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12b11e9f831c 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12b11e9f831c 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:05 2024 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:05 2024 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:05 2024 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 07e9e1a4-7494-4a2b-8016-b9a4a688b970 2024-09-16T13:09:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.export 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.export 2024-09-16T13:09:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.export 2024-09-16T13:09:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.export 2024-09-16T13:09:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.export 2024-09-16T13:09:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.export 2024-09-16T13:09:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_44.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_44.export 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-16T13:09:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T13:09:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-16T13:09:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_45.status 2024-09-16T13:09:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.status 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_45.status 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-16T13:09:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-16T13:09:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 207 2024-09-16T13:09:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.status 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.status 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-16T13:09:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=207 2024-09-16T13:09:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T13:09:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 207 = '' 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T13:09:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 207 = '' 2024-09-16T13:09:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.status 2024-09-16T13:09:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T13:09:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=207 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 207 = 207 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_45': 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c94118dbd3 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c94118dbd3 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:19 2024 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:19 2024 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:19 2024 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 65c4a20f-84d6-4179-a149-1fd4a4176f5d 2024-09-16T13:09:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.export 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.export 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.export 2024-09-16T13:09:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.export 2024-09-16T13:09:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.export 2024-09-16T13:09:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.export 2024-09-16T13:09:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_45.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_45.export 2024-09-16T13:09:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-16T13:09:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T13:09:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-16T13:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T13:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:09:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:09:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:09:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-16T13:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T13:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-16T13:09:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T13:09:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T13:09:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-16T13:09:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_46.status 2024-09-16T13:09:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.status 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_46.status 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-16T13:09:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 209 2024-09-16T13:09:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.status 2024-09-16T13:09:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.status 2024-09-16T13:09:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-16T13:09:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=209 2024-09-16T13:09:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T13:09:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 209 = '' 2024-09-16T13:09:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T13:09:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 209 = '' 2024-09-16T13:09:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.status 2024-09-16T13:09:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T13:09:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=209 2024-09-16T13:09:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 209 = 209 2024-09-16T13:09:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-16T13:09:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-16T13:09:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_46': 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 11409839421f 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.11409839421f 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:25 2024 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:25 2024 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:25 2024 2024-09-16T13:09:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d0e5b05e-776b-457f-b59b-3da9cd242b9e 2024-09-16T13:09:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-16T13:09:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:09:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.export 2024-09-16T13:09:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.export 2024-09-16T13:09:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.export 2024-09-16T13:09:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.export 2024-09-16T13:09:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.export 2024-09-16T13:09:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.export 2024-09-16T13:09:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_46.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_46.export 2024-09-16T13:09:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:09:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-16T13:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:09:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:09:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:09:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:09:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:09:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:09:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:09:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:09:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:09:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:09:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-16T13:09:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-16T13:09:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:09:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:09:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:09:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:09:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:09:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:09:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:09:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:09:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:09:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:09:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-16T13:09:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:09:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:09:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:09:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:09:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_47.status 2024-09-16T13:09:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.status 2024-09-16T13:09:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_47.status 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-16T13:09:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 211 2024-09-16T13:09:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.status 2024-09-16T13:09:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.status 2024-09-16T13:09:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-16T13:09:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:09:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=211 2024-09-16T13:09:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:09:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T13:09:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=114 2024-09-16T13:09:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 211 = 114 2024-09-16T13:09:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T13:09:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=114 2024-09-16T13:09:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 211 = 114 2024-09-16T13:09:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T13:09:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:09:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:09:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:09:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:09:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:09:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 211 = '' 2024-09-16T13:09:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:09:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.status 2024-09-16T13:09:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T13:09:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:09:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:09:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=211 2024-09-16T13:09:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 211 = 211 2024-09-16T13:09:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-16T13:09:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:09:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:09:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_47': 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129be5fc477d 2024-09-16T13:09:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129be5fc477d 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:32 2024 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:32 2024 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:32 2024 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8cae9722-25fd-42a4-9bfa-950a5f46d236 2024-09-16T13:09:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:09:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.export 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.export 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.export 2024-09-16T13:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.export 2024-09-16T13:09:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.export 2024-09-16T13:09:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:09:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.export 2024-09-16T13:09:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_47.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_47.export 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:09:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-16T13:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-16T13:10:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:10:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:10:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:10:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:10:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:10:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:10:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:10:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-16T13:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-16T13:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:10:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:10:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:10:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-16T13:10:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-16T13:10:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:10:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-16T13:10:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-16T13:10:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T13:10:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-16T13:10:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-16T13:10:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T13:10:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:10:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:10:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T13:10:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-16T13:10:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:10:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:10:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:10:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:10:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:10:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_48.status 2024-09-16T13:10:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.status 2024-09-16T13:10:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-16T13:10:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_48.status 2024-09-16T13:10:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-16T13:10:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 213 2024-09-16T13:10:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.status 2024-09-16T13:10:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.status 2024-09-16T13:10:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-16T13:10:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:10:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=213 2024-09-16T13:10:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:10:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T13:10:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 213 = '' 2024-09-16T13:10:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T13:10:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 213 = '' 2024-09-16T13:10:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:10:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.status 2024-09-16T13:10:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T13:10:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=213 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 213 = 213 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:10:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-16T13:10:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_48': 2024-09-16T13:10:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:10:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12c0386d760b 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12c0386d760b 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:47 2024 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:47 2024 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:47 2024 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:10:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:10:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: b0393f6b-edbc-4675-be86-563550c2a642 2024-09-16T13:10:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:10:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.export 2024-09-16T13:10:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.export 2024-09-16T13:10:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.export 2024-09-16T13:10:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.export 2024-09-16T13:10:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.export 2024-09-16T13:10:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.export 2024-09-16T13:10:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_48.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_48.export 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-16T13:10:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:10:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:10:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:10:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-16T13:10:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T13:10:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T13:10:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-16T13:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-16T13:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-16T13:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-16T13:10:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-16T13:10:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T13:10:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-16T13:10:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-16T13:10:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T13:10:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T13:10:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-16T13:10:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:10:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:10:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:10:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:10:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:10:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:10:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-16T13:10:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_49.status 2024-09-16T13:10:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.status 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_49.status 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-16T13:10:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-16T13:10:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 215 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.status 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.status 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-16T13:10:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:10:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=215 2024-09-16T13:10:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T13:10:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 215 = '' 2024-09-16T13:10:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:10:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T13:10:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 215 = '' 2024-09-16T13:10:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:10:11.702 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-16T13:10:11.700+0000 7f09dd39d640 -1 reset not still connected to 0x558b19622820 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.status 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T13:10:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=215 2024-09-16T13:10:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 215 = 215 2024-09-16T13:10:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:10:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-16T13:10:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_49': 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12c05cae2d17 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12c05cae2d17 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:02:58 2024 2024-09-16T13:10:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:02:58 2024 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:02:58 2024 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7e4e667f-21c3-4325-899a-b12ec27f6f23 2024-09-16T13:10:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.export 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.export 2024-09-16T13:10:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.export 2024-09-16T13:10:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.export 2024-09-16T13:10:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.export 2024-09-16T13:10:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.export 2024-09-16T13:10:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_49.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_49.export 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T13:10:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-16T13:10:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-16T13:10:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-16T13:10:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-16T13:10:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-16T13:10:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:10:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-16T13:10:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T13:10:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-16T13:10:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:10:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-16T13:10:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-16T13:10:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-16T13:10:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-16T13:10:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:10:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:10:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-16T13:10:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-16T13:10:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T13:10:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-16T13:10:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-16T13:10:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-16T13:10:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-16T13:10:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_50.status 2024-09-16T13:10:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.status 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_50.status 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-16T13:10:16.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.676591441139844) > weight (0.25) 2024-09-16T13:10:16.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9820226792114574) > weight (0.25) 2024-09-16T13:10:16.896 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9384985147538106) > weight (0.25) 2024-09-16T13:10:16.896 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T13:10:16.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T13:10:16.896 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-16T13:10:16.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:10:16.895+0000 7f8ae55f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38129) UID: 1000 2024-09-16T13:10:17.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:10:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 217 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.status 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.status 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T13:10:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=217 2024-09-16T13:10:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-16T13:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 217 = '' 2024-09-16T13:10:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-16T13:10:17.973 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:10:17.971+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e2311110 2024-09-16T13:10:17.973 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:10:17.971+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e98da750 2024-09-16T13:10:17.973 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-16T13:10:17.971+0000 7fb62d3cd640 -1 reset not still connected to 0x5564e98db520 2024-09-16T13:10:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 217 = '' 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-16T13:10:18.897 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T13:10:18.897 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:10:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:10:18.898 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T13:10:18.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T13:10:18.898 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T13:10:18.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T13:10:18.902 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:10:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 217 = '' 2024-09-16T13:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=117 2024-09-16T13:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 217 = 117 2024-09-16T13:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T13:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-16T13:10:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.status 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T13:10:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T13:10:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=217 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 217 = 217 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:10:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:10:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T13:10:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:10:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_50': 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1140cdfb32bd 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1140cdfb32bd 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:10:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:03:12 2024 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:03:12 2024 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:03:12 2024 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 26bf8c6e-9262-4fd0-975d-7cfaad8dde35 2024-09-16T13:10:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.export 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.export 2024-09-16T13:10:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.export 2024-09-16T13:10:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.export 2024-09-16T13:10:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.export 2024-09-16T13:10:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T13:10:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.export 2024-09-16T13:10:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.cccYcKFUz2/cluster2-mirror-image_50.export /tmp/tmp.cccYcKFUz2/cluster1-mirror-image_50.export 2024-09-16T13:10:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: delete many images' 2024-09-16T13:10:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T13:10:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T13:10:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 13:10:24 TEST: delete many images 2024-09-16T13:10:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 13:10:24 TEST: delete many images 2024-09-16T13:10:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:10:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-16T13:10:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T13:10:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:10:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T13:10:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T13:10:25.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:10:25.581+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=70a8076e-7461-4223-9205-c586ea82527e 2024-09-16T13:10:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-16T13:10:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T13:10:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:10:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T13:10:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:10:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T13:10:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T13:10:27.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:10:27.583+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=5d32e8f9-d6e5-4ee3-9c1a-b169f80cfbb0 2024-09-16T13:10:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-16T13:10:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T13:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:10:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T13:10:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T13:10:29.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:29.633+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbee769c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:29.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:29.633+0000 7fcf8fff7640 -1 rbd::mirror::ImageReplayer: 0x558bbbb10500 [3/fa91e86c-1fbe-49d8-980f-ba4716f8e375] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:10:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-16T13:10:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T13:10:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T13:10:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:10:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T13:10:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T13:10:31.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:31.681+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbee76b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:31.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:31.681+0000 7fcf8fff7640 -1 rbd::mirror::ImageReplayer: 0x558bbbb10280 [3/9f626ce6-5b6e-44bf-be4c-04da712913b0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-16T13:10:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T13:10:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:10:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T13:10:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:10:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T13:10:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T13:10:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-16T13:10:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-16T13:10:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:10:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-16T13:10:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T13:10:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:10:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T13:10:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T13:10:34.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:10:34.808+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2b271520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:34.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:10:34.808+0000 7f48529f7640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4a00 [3/44d2ade4-d0ac-4f1d-a249-9712f1c710b8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-16T13:10:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T13:10:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:10:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T13:10:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:10:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T13:10:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T13:10:36.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:10:36.765+0000 7f65093f7640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=e7069a5d-0e13-4d1b-976c-04e9409a6d09 2024-09-16T13:10:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-16T13:10:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T13:10:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:10:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T13:10:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:10:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T13:10:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T13:10:38.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:38.824+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbf34b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:38.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:38.824+0000 7fcf907f8640 -1 rbd::mirror::ImageReplayer: 0x558bbbb10f00 [3/470fe0b0-f9fc-45ce-91f1-79e5c3c5fc0e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-16T13:10:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T13:10:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:10:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T13:10:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T13:10:40.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:10:40.783+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=aa2d39b0-ecda-4092-b3c6-43b942830859 2024-09-16T13:10:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-16T13:10:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-16T13:10:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:10:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-16T13:10:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T13:10:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:10:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T13:10:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T13:10:42.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:10:42.788+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=a9b282ed-f577-495d-bb2c-a0ac91558148 2024-09-16T13:10:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-16T13:10:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-16T13:10:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:10:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_11 2024-09-16T13:10:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:10:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-16T13:10:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-16T13:10:44.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:10:44.842+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2b274820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:44.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:10:44.842+0000 7f48529f7640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d5900 [3/000b0e94-a5bd-4706-9afd-99ff1b8bf462] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:44.846 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T13:10:44.843+0000 7f6068a65640 -1 reset not still connected to 0x55b6ee4f0d00 2024-09-16T13:10:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-16T13:10:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_12 2024-09-16T13:10:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:10:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-16T13:10:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-16T13:10:46.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:10:46.846+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583deaed860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:46.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:10:46.846+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583de7cac80 [3/42c7f0d4-ea22-4303-82c7-5d306ec75295] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-16T13:10:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_13 2024-09-16T13:10:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:10:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-16T13:10:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-16T13:10:49.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:49.024+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a6465040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:49.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:49.024+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a2dc8280 [3/451fa1a3-9713-4bdb-894c-abccb64adbc2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:10:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-16T13:10:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_14 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:10:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-16T13:10:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-16T13:10:50.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:50.851+0000 7fcf89feb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbec89520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:50.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:50.851+0000 7fcf907f8640 -1 rbd::mirror::ImageReplayer: 0x558bbf0a9680 [3/f3868c68-fe40-4444-8720-528d92e98de6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:10:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-16T13:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_15 2024-09-16T13:10:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:10:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-16T13:10:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-16T13:10:52.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:52.873+0000 7fae70dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a64729c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:52.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:52.873+0000 7fae77df8640 -1 rbd::mirror::ImageReplayer: 0x5627a2dc8780 [3/f65d50c0-9e44-4158-8c3d-91b35d21fe9a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-16T13:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_16 2024-09-16T13:10:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:10:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-16T13:10:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-16T13:10:54.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:54.863+0000 7fae70dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a6473ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:54.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:10:54.863+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a6050000 [3/f77bbb21-fc88-4735-8cf0-73c5d5cc573f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-16T13:10:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_17 2024-09-16T13:10:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:10:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-16T13:10:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-16T13:10:56.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:10:56.888+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dc0f7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:56.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:10:56.888+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583de7cb900 [3/105a819b-f484-4d1d-a16e-4f48810acd47] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-16T13:10:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:10:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-16T13:10:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:10:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:10:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:10:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_18 2024-09-16T13:10:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:10:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-16T13:10:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-16T13:10:58.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:58.889+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbf250d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:10:58.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:10:58.889+0000 7fcf907f8640 -1 rbd::mirror::ImageReplayer: 0x558bbf0a8780 [3/d03e92a5-c8e6-4505-aae2-e8c01ac781fc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:10:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:10:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-16T13:11:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_19 2024-09-16T13:11:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:11:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-16T13:11:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-16T13:11:00.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:00.899+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583deaeda00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:00.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:00.899+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583de9bea00 [3/7c010caf-1b79-4274-b3dd-76aca299a0a4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:11:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-16T13:11:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-16T13:11:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:11:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_20 2024-09-16T13:11:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:11:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-16T13:11:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-16T13:11:03.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:03.114+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dc0f7d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:03.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:03.114+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583db3ee780 [3/897043d8-7460-4263-8c01-2902509580ba] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-16T13:11:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_21 2024-09-16T13:11:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:11:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-16T13:11:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-16T13:11:04.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:04.923+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583debe7d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:04.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:04.923+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583deb98280 [3/5ec8e0c7-23d8-4f56-a4da-5c7e8d8a4880] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:05.826 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5321323022886298) > weight (0.25) 2024-09-16T13:11:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.35895686353162093) > weight (0.25) 2024-09-16T13:11:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2505461418675715) > weight (0.25) 2024-09-16T13:11:05.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5294122821716383) > weight (0.25) 2024-09-16T13:11:05.827 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-16T13:11:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-16T13:11:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-16T13:11:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_22 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:11:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-16T13:11:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-16T13:11:06.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:06.938+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a63756c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:06.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:06.938+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a2dc9180 [3/4135e1aa-5a4e-4f22-a4a4-d979c6386191] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-16T13:11:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-16T13:11:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:11:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-16T13:11:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_23 2024-09-16T13:11:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:11:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-16T13:11:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-16T13:11:08.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:08.944+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dea6ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:08.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:08.944+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583db3ee280 [3/deee2aeb-a92c-4e22-ab10-fb00148a0183] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-16T13:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_24 2024-09-16T13:11:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:11:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-16T13:11:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-16T13:11:11.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:11.161+0000 7f484c9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e4ddd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:11.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:11.161+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d5680 [3/d84f0610-c1af-4671-8b4a-aeb184619555] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-16T13:11:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-16T13:11:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_25 2024-09-16T13:11:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:11:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-16T13:11:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-16T13:11:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:12.962+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a6375d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:12.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:12.962+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a2dc9b80 [3/dde89dcd-ca3e-4c90-a12d-78663eac03bf] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-16T13:11:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-16T13:11:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:11:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_26 2024-09-16T13:11:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:11:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-16T13:11:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-16T13:11:14.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:14.955+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dea6ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:14.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:14.955+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583dea53b80 [3/3b3040ad-963e-4bf9-8b3f-18deb1f76778] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-16T13:11:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-16T13:11:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:11:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_27 2024-09-16T13:11:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:11:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-16T13:11:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-16T13:11:16.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:16.973+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e4dd380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:16.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:16.973+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d5400 [3/6e695d31-7fbb-4a6d-bf1c-da34f746931f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:11:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-16T13:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-16T13:11:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_28 2024-09-16T13:11:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:11:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-16T13:11:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-16T13:11:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:19.167+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dea6a820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:19.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:19.167+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583db3efb80 [3/ed51faf8-e2c3-44a4-9d09-787be9fd78f9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:11:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-16T13:11:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-16T13:11:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:11:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_29 2024-09-16T13:11:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:11:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-16T13:11:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-16T13:11:21.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:21.172+0000 7f484c9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2be74000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:21.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:21.172+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4280 [3/21593e92-6a4d-4963-8412-532dd99efda2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:11:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-16T13:11:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-16T13:11:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:11:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_30 2024-09-16T13:11:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:11:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-16T13:11:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-16T13:11:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:22.985+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e58c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:22.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:22.985+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2d652000 [3/795f020a-5b0a-43d5-a652-615b46e9624a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-16T13:11:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_31 2024-09-16T13:11:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:11:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-16T13:11:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-16T13:11:24.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:24.807+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583de6a2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:24.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:24.807+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583db3eef00 [3/eeb2f380-5802-4457-936c-f678d743f57e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:11:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-16T13:11:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-16T13:11:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_32 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:11:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-16T13:11:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-16T13:11:26.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:26.818+0000 7f90621eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dea6bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:26.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:26.818+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583deb98f00 [3/7bac516b-dc94-4b85-9cd0-8bcb96179e9a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-16T13:11:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-16T13:11:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:11:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_33 2024-09-16T13:11:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:11:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-16T13:11:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-16T13:11:28.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:28.815+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a539d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:28.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:28.815+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a2dc9400 [3/686e94e1-8f41-41d6-998f-9112a95df1d3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-16T13:11:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-16T13:11:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:11:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_34 2024-09-16T13:11:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:11:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-16T13:11:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-16T13:11:30.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:30.825+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a6472d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:30.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:30.825+0000 7fae77df8640 -1 rbd::mirror::ImageReplayer: 0x5627a629b900 [3/321f4d24-c15c-4025-9b78-f2e2dd17716a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-16T13:11:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_35 2024-09-16T13:11:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:11:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-16T13:11:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-16T13:11:32.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:32.839+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbee77040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:32.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:32.839+0000 7fcf8fff7640 -1 rbd::mirror::ImageReplayer: 0x558bbf29b900 [3/1bac1448-c694-4f80-90b6-6bd1f7068529] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:32.903 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8198556612021294) > weight (0.25) 2024-09-16T13:11:32.903 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T13:11:32.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T13:11:32.904 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T13:11:32.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:11:32.902+0000 7f3bfbbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 61808) UID: 1000 2024-09-16T13:11:33.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:11:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-16T13:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_36 2024-09-16T13:11:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:11:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-16T13:11:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-16T13:11:34.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:34.833+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e58c820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:34.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:34.833+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4c80 [3/573c8bbb-eb7d-47c7-a3f5-dd7fbc895516] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-16T13:11:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_37 2024-09-16T13:11:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:11:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-16T13:11:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-16T13:11:36.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:36.846+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583deaed860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:36.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:36.846+0000 7f90689f8640 -1 rbd::mirror::ImageReplayer: 0x5583de7ca280 [3/da41c0e4-7e5d-4612-8458-a799307ed96f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:11:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-16T13:11:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_38 2024-09-16T13:11:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:11:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-16T13:11:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-16T13:11:38.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:38.856+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a6472ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:38.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:38.856+0000 7fae77df8640 -1 rbd::mirror::ImageReplayer: 0x5627a63fa000 [3/721e34a7-a4b2-4332-a181-ccded18e8696] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-16T13:11:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-16T13:11:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:11:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_39 2024-09-16T13:11:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:11:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-16T13:11:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-16T13:11:40.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:40.885+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583dea6b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:40.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:11:40.885+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583db3ef900 [3/fe577bd1-b7b2-46e6-adee-f694b7fb7801] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-16T13:11:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-16T13:11:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:11:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-16T13:11:41.904 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T13:11:41.904 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:11:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:11:41.905 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T13:11:41.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T13:11:41.905 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T13:11:41.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T13:11:41.908 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-16T13:11:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_40 2024-09-16T13:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-16T13:11:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-16T13:11:43.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:11:43.026+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=a44aec3e-055a-42d9-a75a-50b8b1713251 2024-09-16T13:11:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-16T13:11:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_41 2024-09-16T13:11:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:11:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-16T13:11:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-16T13:11:44.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:44.881+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e58c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:44.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:44.881+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4780 [3/3e6603df-4a85-45e2-92bc-3718de0d61a1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-16T13:11:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_42 2024-09-16T13:11:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:11:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-16T13:11:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-16T13:11:46.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:11:46.845+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=362a7f1e-b75e-484c-bd35-6308692dbcfe 2024-09-16T13:11:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-16T13:11:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_43 2024-09-16T13:11:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:11:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-16T13:11:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-16T13:11:48.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:48.898+0000 7fcf89feb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbee771e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:48.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:48.898+0000 7fcf8fff7640 -1 rbd::mirror::ImageReplayer: 0x558bbf0a9180 [3/89399335-667d-40ff-83af-2f8d80ec3800] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:49.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:49.103+0000 7fcf897ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558bbf387d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:49.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:11:49.103+0000 7fcf907f8640 -1 rbd::mirror::ImageReplayer: 0x558bbf0a9180 [3/89399335-667d-40ff-83af-2f8d80ec3800] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:11:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-16T13:11:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-16T13:11:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:11:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_44 2024-09-16T13:11:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:11:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-16T13:11:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-16T13:11:50.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:11:50.844+0000 7f65093f7640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=07e9e1a4-7494-4a2b-8016-b9a4a688b970 2024-09-16T13:11:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-16T13:11:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_45 2024-09-16T13:11:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:11:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-16T13:11:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-16T13:11:52.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:11:52.862+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=65c4a20f-84d6-4179-a149-1fd4a4176f5d 2024-09-16T13:11:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-16T13:11:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-16T13:11:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_46 2024-09-16T13:11:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:11:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-16T13:11:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-16T13:11:54.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:54.901+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e58cb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:54.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:54.901+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4f00 [3/d0e5b05e-776b-457f-b59b-3da9cd242b9e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:54.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:54.944+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e687380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:54.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:11:54.944+0000 7f48529f7640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4f00 [3/d0e5b05e-776b-457f-b59b-3da9cd242b9e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-16T13:11:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-16T13:11:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:11:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_47 2024-09-16T13:11:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:11:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-16T13:11:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-16T13:11:56.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:11:56.880+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=8cae9722-25fd-42a4-9bfa-950a5f46d236 2024-09-16T13:11:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:11:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-16T13:11:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:11:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-16T13:11:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:11:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:11:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:11:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_48 2024-09-16T13:11:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:11:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-16T13:11:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-16T13:11:58.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:58.934+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a539d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:11:58.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:11:58.934+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a6050a00 [3/b0393f6b-edbc-4675-be86-563550c2a642] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:11:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:11:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:11:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:11:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:11:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-16T13:12:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:12:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:12:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_49 2024-09-16T13:12:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:12:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-16T13:12:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-16T13:12:00.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:12:00.902+0000 7f6509bf8640 -1 rbd::mirror::ImageMap: 0x55a7a945a900 handle_peer_ack_remove: failed to remove global_image_id=7e4e667f-21c3-4325-899a-b12ec27f6f23 2024-09-16T13:12:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:12:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-16T13:12:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-16T13:12:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:12:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:12:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_50 2024-09-16T13:12:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:12:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-16T13:12:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-16T13:12:03.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:12:03.152+0000 7f484c1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558f2e6876c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:12:03.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:12:03.152+0000 7f48531f8640 -1 rbd::mirror::ImageReplayer: 0x558f2b1d4500 [3/26bf8c6e-9262-4fd0-975d-7cfaad8dde35] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:12:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T13:12:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-16T13:12:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T13:12:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T13:12:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 13:12:03 TEST: image deletions should propagate 2024-09-16T13:12:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 13:12:03 TEST: image deletions should propagate 2024-09-16T13:12:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-16T13:12:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=0 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:12:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-16T13:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-16T13:12:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=6 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = 0 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 -ne 0 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ break 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 -eq 0 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_count=6 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-16T13:12:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:12:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:12:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:12:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:12:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-16T13:12:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-16T13:12:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=6 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 = 0 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 6 -ne 6 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 2 -ne 1 2024-09-16T13:12:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 30 2024-09-16T13:12:08.534 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T13:12:08.531+0000 7feb3baab640 -1 reset not still connected to 0x561da0ea6b60 2024-09-16T13:12:08.534 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T13:12:08.531+0000 7feb3baab640 -1 reset not still connected to 0x561da0ea7450 2024-09-16T13:12:08.534 INFO:tasks.ceph.cluster1.osd.1.smithi177.stderr:2024-09-16T13:12:08.532+0000 7feb3baab640 -1 reset not still connected to 0x561da0ea7ba0 2024-09-16T13:12:10.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.28168990477863476) > weight (0.25) 2024-09-16T13:12:10.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9948692428168704) > weight (0.25) 2024-09-16T13:12:10.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3226230005420996) > weight (0.25) 2024-09-16T13:12:10.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.883290202430067) > weight (0.25) 2024-09-16T13:12:10.828 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-16T13:12:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:12:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-16T13:12:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-16T13:12:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=0 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-16T13:12:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-16T13:12:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-16T13:12:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T13:12:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-16T13:12:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-16T13:12:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T13:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-16T13:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-16T13:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-16T13:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-16T13:12:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:12:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:12:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T13:12:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:12:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:12:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T13:12:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:12:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:12:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T13:12:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:12:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:12:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:12:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T13:12:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:12:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:12:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:12:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T13:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T13:12:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:12:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:12:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T13:12:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:12:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T13:12:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T13:12:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:12:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:12:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T13:12:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T13:12:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:12:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:12:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T13:12:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:12:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:12:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T13:12:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:12:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T13:12:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:12:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:12:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T13:12:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:12:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T13:12:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:12:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T13:12:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T13:12:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:12:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:12:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T13:12:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:12:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T13:12:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:12:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:12:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T13:12:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T13:12:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:12:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:12:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_11 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T13:12:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T13:12:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T13:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_11 2024-09-16T13:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T13:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T13:12:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T13:12:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_12 2024-09-16T13:12:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T13:12:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T13:12:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_12 2024-09-16T13:12:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T13:12:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T13:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T13:12:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_13 2024-09-16T13:12:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T13:12:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T13:12:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-16T13:12:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_13 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T13:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T13:12:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_14 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T13:12:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T13:12:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-16T13:12:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_14 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T13:12:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_15 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T13:12:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T13:12:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_15 2024-09-16T13:12:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T13:12:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:12:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_16 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T13:12:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T13:12:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_16 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T13:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T13:12:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T13:12:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_17 2024-09-16T13:12:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T13:12:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T13:12:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_17 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T13:12:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-16T13:12:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_18 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T13:12:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T13:12:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-16T13:12:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_18 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T13:12:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_19 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T13:12:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T13:12:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_19 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T13:12:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T13:12:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_20 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T13:12:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T13:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_20 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T13:12:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T13:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T13:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_21 2024-09-16T13:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T13:12:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T13:12:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-16T13:12:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_21 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T13:12:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T13:12:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T13:12:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_22 2024-09-16T13:12:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T13:12:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T13:12:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-16T13:12:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_22 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T13:12:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:12:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_23 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T13:12:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T13:12:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-16T13:12:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-16T13:12:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_23 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T13:12:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T13:12:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_24 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T13:12:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T13:12:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-16T13:12:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_24 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T13:12:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T13:12:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T13:12:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_25 2024-09-16T13:12:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T13:12:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T13:12:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_25 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T13:12:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_26 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T13:12:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T13:12:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_26 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T13:12:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_27 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T13:12:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T13:12:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T13:12:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_27 2024-09-16T13:12:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T13:12:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-16T13:12:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_28 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T13:12:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T13:12:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-16T13:12:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_28 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T13:12:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T13:12:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_29 2024-09-16T13:12:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T13:12:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T13:12:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_29 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T13:12:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T13:12:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T13:12:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_30 2024-09-16T13:12:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T13:12:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T13:12:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_30 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T13:12:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T13:12:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_31 2024-09-16T13:12:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T13:12:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T13:12:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_31 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T13:12:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T13:12:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_32 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T13:12:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T13:12:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_32 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T13:12:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_33 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T13:12:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T13:12:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_33 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T13:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:12:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_34 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T13:12:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T13:12:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-16T13:12:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-16T13:12:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_34 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T13:12:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_35 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T13:12:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T13:12:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T13:12:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_35 2024-09-16T13:12:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T13:12:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T13:12:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_36 2024-09-16T13:12:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T13:12:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T13:12:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-16T13:12:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-16T13:12:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_36 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T13:12:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T13:12:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_37 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T13:12:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T13:12:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_37 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T13:12:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T13:12:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_38 2024-09-16T13:12:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T13:12:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T13:12:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-16T13:12:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_38 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T13:12:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T13:12:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_39 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T13:12:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T13:12:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_39 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T13:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T13:12:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_40 2024-09-16T13:12:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T13:12:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T13:12:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T13:12:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_40 2024-09-16T13:12:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T13:12:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_41 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T13:12:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T13:12:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_41 2024-09-16T13:12:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T13:12:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_42 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T13:12:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T13:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_42 2024-09-16T13:12:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T13:12:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-16T13:12:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_43 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T13:12:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T13:12:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-16T13:12:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-16T13:12:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_43 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T13:12:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T13:12:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_44 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T13:12:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T13:12:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-16T13:12:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_44 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T13:12:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_45 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T13:12:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T13:12:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-16T13:12:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-16T13:12:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_45 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T13:12:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T13:12:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T13:12:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_46 2024-09-16T13:12:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T13:12:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T13:12:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-16T13:12:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_46 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T13:12:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T13:12:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_47 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T13:12:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T13:12:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-16T13:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_47 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T13:12:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T13:12:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_48 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T13:12:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T13:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_48 2024-09-16T13:12:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T13:12:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T13:12:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_49 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T13:12:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T13:12:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_49 2024-09-16T13:12:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T13:12:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-16T13:12:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_50 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T13:12:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_50 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T13:12:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 13:12:42 TEST: delete images during bootstrap 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 13:12:42 TEST: delete images during bootstrap 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=image 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-16T13:12:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd: mirroring is already configured for image mode 2024-09-16T13:12:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-16T13:12:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=image 2024-09-16T13:12:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-16T13:12:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd: mirroring is already configured for image mode 2024-09-16T13:12:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1 2024-09-16T13:12:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-16T13:12:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-16T13:12:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=cluster1 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-16T13:12:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=test 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:12:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:12:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2024-09-16T13:12:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-16T13:12:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:12:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T13:12:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:12:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 18e3aa6d11f8 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.18e3aa6d11f8 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:12:42 2024 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:12:42 2024 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:12:42 2024 2024-09-16T13:12:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:12:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:12:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 00e90898-f701-4457-bdd5-f6169df639c4 2024-09-16T13:12:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:12:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T13:12:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:12:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:12:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:12:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:12:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 28808192 bytes 2024-09-16T13:12:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 56378368 bytes 2024-09-16T13:12:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 150106112 bytes 2024-09-16T13:12:44.565 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T13:12:44.562+0000 7f9655eab640 -1 reset not still connected to 0x56310e968ea0 2024-09-16T13:12:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 223439872 bytes 2024-09-16T13:12:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 278263808 bytes 2024-09-16T13:12:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 410289152 bytes 2024-09-16T13:12:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 438127616 bytes 2024-09-16T13:12:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 443066368 bytes 2024-09-16T13:12:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 462140416 bytes 2024-09-16T13:12:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 528983040 bytes 2024-09-16T13:12:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 614604800 bytes 2024-09-16T13:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 648285184 bytes 2024-09-16T13:12:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 654860288 bytes 2024-09-16T13:12:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 736710656 bytes 2024-09-16T13:12:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 799799296 bytes 2024-09-16T13:12:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 878668800 bytes 2024-09-16T13:12:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 884867072 bytes 2024-09-16T13:12:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 910121984 bytes 2024-09-16T13:12:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 993575936 bytes 2024-09-16T13:12:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1025257472 bytes 2024-09-16T13:12:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1042946048 bytes 2024-09-16T13:12:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1073861632 bytes 2024-09-16T13:12:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1080763392 bytes 2024-09-16T13:12:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1140445184 bytes 2024-09-16T13:12:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1209745408 bytes 2024-09-16T13:12:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1214844928 bytes 2024-09-16T13:12:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1263603712 bytes 2024-09-16T13:12:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1332692992 bytes 2024-09-16T13:12:48.909 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6136223437553004) > weight (0.25) 2024-09-16T13:12:48.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.33712498101662025) > weight (0.25) 2024-09-16T13:12:48.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9858522192452776) > weight (0.25) 2024-09-16T13:12:48.910 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4399192321364761) > weight (0.25) 2024-09-16T13:12:48.910 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-16T13:12:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1387043840 bytes 2024-09-16T13:12:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1434756096 bytes 2024-09-16T13:12:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1470094336 bytes 2024-09-16T13:12:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1519937536 bytes 2024-09-16T13:12:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1553996800 bytes 2024-09-16T13:12:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1602973696 bytes 2024-09-16T13:12:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1655222272 bytes 2024-09-16T13:12:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1698862080 bytes 2024-09-16T13:12:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1735671808 bytes 2024-09-16T13:12:50.409 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-16T13:12:50.407+0000 7f9655eab640 -1 reset not still connected to 0x56310e968ea0 2024-09-16T13:12:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1771762688 bytes 2024-09-16T13:12:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1800482816 bytes 2024-09-16T13:12:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1834298368 bytes 2024-09-16T13:12:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1875570688 bytes 2024-09-16T13:12:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1938648064 bytes 2024-09-16T13:12:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1973185536 bytes 2024-09-16T13:12:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1990009856 bytes 2024-09-16T13:12:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2065206272 bytes 2024-09-16T13:12:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2127011840 bytes 2024-09-16T13:12:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2130902016 bytes 2024-09-16T13:12:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2203162624 bytes 2024-09-16T13:12:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2317108224 bytes 2024-09-16T13:12:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2346458112 bytes 2024-09-16T13:12:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2411856896 bytes 2024-09-16T13:12:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2447606784 bytes 2024-09-16T13:12:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2520797184 bytes 2024-09-16T13:12:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2590174208 bytes 2024-09-16T13:12:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2635147264 bytes 2024-09-16T13:12:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2641098752 bytes 2024-09-16T13:12:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2681459712 bytes 2024-09-16T13:12:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2751659008 bytes 2024-09-16T13:12:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2809680896 bytes 2024-09-16T13:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2878223360 bytes 2024-09-16T13:12:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2921970688 bytes 2024-09-16T13:12:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 3006117888 bytes 2024-09-16T13:12:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3044980736 bytes 2024-09-16T13:12:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3069229056 bytes 2024-09-16T13:12:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3098627072 bytes 2024-09-16T13:12:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3156638720 bytes 2024-09-16T13:12:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:12:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:12:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:12:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:12:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129b2cb38f50 2024-09-16T13:12:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:12:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 present 129b2cb38f50 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b2cb38f50 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:12:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:12:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:12:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:12:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b2cb38f50 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b2cb38f50 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b2cb38f50 -o 129b2cb38f50 = 129b2cb38f50 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:12:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-16T13:12:56.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:12:56.868+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2aad800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2024-09-16T13:12:56.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:12:56.868+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2aad800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2024-09-16T13:13:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:13:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:13:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T13:13:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:13:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:13:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:13:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-16T13:13:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T13:13:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:13:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-16T13:13:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:13:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T13:13:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-16T13:13:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T13:13:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T13:13:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:13:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:13:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:13:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T13:13:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:13:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T13:13:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 18f540f1ee12 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.18f540f1ee12 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:13:26 2024 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:13:26 2024 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:13:26 2024 2024-09-16T13:13:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:13:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:13:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7f77edaf-cac1-4c2f-b9a0-028258811174 2024-09-16T13:13:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:13:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T13:13:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:13:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-16T13:13:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:13:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:13:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 77766656 bytes 2024-09-16T13:13:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 160104448 bytes 2024-09-16T13:13:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 214968320 bytes 2024-09-16T13:13:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 229796864 bytes 2024-09-16T13:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 313773056 bytes 2024-09-16T13:13:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 355637248 bytes 2024-09-16T13:13:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 396452864 bytes 2024-09-16T13:13:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 450512896 bytes 2024-09-16T13:13:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 496560128 bytes 2024-09-16T13:13:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 524922880 bytes 2024-09-16T13:13:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 616898560 bytes 2024-09-16T13:13:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 639993856 bytes 2024-09-16T13:13:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 659107840 bytes 2024-09-16T13:13:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 672632832 bytes 2024-09-16T13:13:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 688188416 bytes 2024-09-16T13:13:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 706859008 bytes 2024-09-16T13:13:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 723362816 bytes 2024-09-16T13:13:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 763066368 bytes 2024-09-16T13:13:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 790281216 bytes 2024-09-16T13:13:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 797382656 bytes 2024-09-16T13:13:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 846840832 bytes 2024-09-16T13:13:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 926334976 bytes 2024-09-16T13:13:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 979858432 bytes 2024-09-16T13:13:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1030189056 bytes 2024-09-16T13:13:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1069818880 bytes 2024-09-16T13:13:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1119160320 bytes 2024-09-16T13:13:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1152515072 bytes 2024-09-16T13:13:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1167063040 bytes 2024-09-16T13:13:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1217188864 bytes 2024-09-16T13:13:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1217824768 bytes 2024-09-16T13:13:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1269223424 bytes 2024-09-16T13:13:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1314674688 bytes 2024-09-16T13:13:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1408978944 bytes 2024-09-16T13:13:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1443277824 bytes 2024-09-16T13:13:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1456488448 bytes 2024-09-16T13:13:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1492250624 bytes 2024-09-16T13:13:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1545923584 bytes 2024-09-16T13:13:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1609437184 bytes 2024-09-16T13:13:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1683083264 bytes 2024-09-16T13:13:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1747717120 bytes 2024-09-16T13:13:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1812890624 bytes 2024-09-16T13:13:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1813514240 bytes 2024-09-16T13:13:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1840527360 bytes 2024-09-16T13:13:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1880269824 bytes 2024-09-16T13:13:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1942005760 bytes 2024-09-16T13:13:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1998552064 bytes 2024-09-16T13:13:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2021007360 bytes 2024-09-16T13:13:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2048584704 bytes 2024-09-16T13:13:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2110402560 bytes 2024-09-16T13:13:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2161601536 bytes 2024-09-16T13:13:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2228325376 bytes 2024-09-16T13:13:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2253200384 bytes 2024-09-16T13:13:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2294574080 bytes 2024-09-16T13:13:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2317656064 bytes 2024-09-16T13:13:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2318270464 bytes 2024-09-16T13:13:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2342903808 bytes 2024-09-16T13:13:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2382648320 bytes 2024-09-16T13:13:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2418907136 bytes 2024-09-16T13:13:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2512022528 bytes 2024-09-16T13:13:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2537838592 bytes 2024-09-16T13:13:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2573022208 bytes 2024-09-16T13:13:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2608933888 bytes 2024-09-16T13:13:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2649729024 bytes 2024-09-16T13:13:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2673419264 bytes 2024-09-16T13:13:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2727342080 bytes 2024-09-16T13:13:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2743669760 bytes 2024-09-16T13:13:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2804880384 bytes 2024-09-16T13:13:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2883442688 bytes 2024-09-16T13:13:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2942798848 bytes 2024-09-16T13:13:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3010163712 bytes 2024-09-16T13:13:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3102928896 bytes 2024-09-16T13:13:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3178527744 bytes 2024-09-16T13:13:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3202949120 bytes 2024-09-16T13:13:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3246461952 bytes 2024-09-16T13:13:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3350318080 bytes 2024-09-16T13:13:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3411899392 bytes 2024-09-16T13:13:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:13:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:13:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:13:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:13:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129bad5af77c 2024-09-16T13:13:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:13:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:13:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 present 129bad5af77c 2024-09-16T13:13:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129bad5af77c 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:13:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129bad5af77c 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129bad5af77c 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129bad5af77c -o 129bad5af77c = 129bad5af77c 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:13:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-16T13:13:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-16T13:13:41.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:13:41.683+0000 7f7b87fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd293fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2024-09-16T13:13:41.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:13:41.683+0000 7f7b87fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd293fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2024-09-16T13:13:42.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8508314711768973) > weight (0.25) 2024-09-16T13:13:42.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7371282133600847) > weight (0.25) 2024-09-16T13:13:42.830 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T13:13:42.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T13:13:42.830 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T13:13:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:13:42.829+0000 7fcf90ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 50250) UID: 1000 2024-09-16T13:13:43.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:13:43.831 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T13:13:43.831 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:13:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:13:43.832 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T13:13:43.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T13:13:43.832 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T13:13:43.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T13:13:43.836 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-16T13:13:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:13:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:13:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T13:13:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:13:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:13:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:13:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:13:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-16T13:13:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T13:13:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:13:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T13:13:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:13:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:13:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T13:13:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:13:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:13:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-16T13:13:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T13:13:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T13:14:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:14:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:14:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T13:14:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:14:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1911cf57a02d 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1911cf57a02d 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:14:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:14:09 2024 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:14:09 2024 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:14:09 2024 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: defc9186-8204-4b4a-9cb1-acda146f624d 2024-09-16T13:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:14:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T13:14:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-16T13:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:14:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 601088 bytes 2024-09-16T13:14:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 25492480 bytes 2024-09-16T13:14:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 75994112 bytes 2024-09-16T13:14:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 80658432 bytes 2024-09-16T13:14:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 91105280 bytes 2024-09-16T13:14:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 170593280 bytes 2024-09-16T13:14:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 195246080 bytes 2024-09-16T13:14:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 218330112 bytes 2024-09-16T13:14:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 233926656 bytes 2024-09-16T13:14:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 277886976 bytes 2024-09-16T13:14:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 326085632 bytes 2024-09-16T13:14:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 340518912 bytes 2024-09-16T13:14:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 452157440 bytes 2024-09-16T13:14:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 501050368 bytes 2024-09-16T13:14:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 541233152 bytes 2024-09-16T13:14:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 606295040 bytes 2024-09-16T13:14:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 643318784 bytes 2024-09-16T13:14:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 669409280 bytes 2024-09-16T13:14:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 764086272 bytes 2024-09-16T13:14:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 855565312 bytes 2024-09-16T13:14:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 887353344 bytes 2024-09-16T13:14:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 956139520 bytes 2024-09-16T13:14:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1020312576 bytes 2024-09-16T13:14:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1086748672 bytes 2024-09-16T13:14:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1122408448 bytes 2024-09-16T13:14:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1135434752 bytes 2024-09-16T13:14:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1173191680 bytes 2024-09-16T13:14:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1176845312 bytes 2024-09-16T13:14:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1219250176 bytes 2024-09-16T13:14:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1246390272 bytes 2024-09-16T13:14:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1258079232 bytes 2024-09-16T13:14:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1305413632 bytes 2024-09-16T13:14:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1341542400 bytes 2024-09-16T13:14:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1368165376 bytes 2024-09-16T13:14:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1376865280 bytes 2024-09-16T13:14:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1385969664 bytes 2024-09-16T13:14:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1443246080 bytes 2024-09-16T13:14:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1502124032 bytes 2024-09-16T13:14:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1566246912 bytes 2024-09-16T13:14:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1616859136 bytes 2024-09-16T13:14:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1684373504 bytes 2024-09-16T13:14:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1706411008 bytes 2024-09-16T13:14:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1756222464 bytes 2024-09-16T13:14:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1797145600 bytes 2024-09-16T13:14:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1833079808 bytes 2024-09-16T13:14:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1955225600 bytes 2024-09-16T13:14:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2049463296 bytes 2024-09-16T13:14:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2102623232 bytes 2024-09-16T13:14:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2122025984 bytes 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:14:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:14:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129beda4211e 2024-09-16T13:14:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:14:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:14:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 present 129beda4211e 2024-09-16T13:14:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129beda4211e 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:14:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129beda4211e 2024-09-16T13:14:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129beda4211e 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129beda4211e -o 129beda4211e = 129beda4211e 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-16T13:14:18.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:14:18.773+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd5ec2800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2024-09-16T13:14:18.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:14:18.773+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd5ec2800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2024-09-16T13:14:19.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:14:19.974+0000 7f90619ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583de8324e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:14:19.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:14:19.974+0000 7f90681f7640 -1 rbd::mirror::ImageReplayer: 0x5583deb99680 [3/defc9186-8204-4b4a-9cb1-acda146f624d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:14:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:14:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T13:14:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:14:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T13:14:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:14:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:14:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:14:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T13:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-16T13:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T13:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T13:14:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:14:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:14:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-16T13:14:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:14:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:14:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T13:14:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-16T13:14:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T13:14:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T13:14:43.911 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T13:14:43.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T13:14:43.912 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T13:14:43.912 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T13:14:43.912 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:14:43.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:14:43.910+0000 7f650a3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58564) UID: 1000 2024-09-16T13:14:44.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:14:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:14:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2024-09-16T13:14:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-16T13:14:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:14:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T13:14:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:14:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 192515d19861 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.192515d19861 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:14:45 2024 2024-09-16T13:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:14:45 2024 2024-09-16T13:14:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:14:45 2024 2024-09-16T13:14:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:14:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:14:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ffdb3598-6db9-408b-b739-9883de3dc98a 2024-09-16T13:14:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:14:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T13:14:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:14:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:14:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:14:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:14:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 51796992 bytes 2024-09-16T13:14:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 89195520 bytes 2024-09-16T13:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 169289728 bytes 2024-09-16T13:14:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 212630528 bytes 2024-09-16T13:14:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 224198656 bytes 2024-09-16T13:14:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 289638400 bytes 2024-09-16T13:14:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 322556928 bytes 2024-09-16T13:14:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 365048832 bytes 2024-09-16T13:14:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 442682368 bytes 2024-09-16T13:14:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 472296448 bytes 2024-09-16T13:14:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 510179328 bytes 2024-09-16T13:14:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 572332032 bytes 2024-09-16T13:14:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 612701184 bytes 2024-09-16T13:14:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 675249152 bytes 2024-09-16T13:14:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 736507904 bytes 2024-09-16T13:14:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 807315456 bytes 2024-09-16T13:14:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 868365312 bytes 2024-09-16T13:14:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 869901312 bytes 2024-09-16T13:14:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:14:50.014 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T13:14:50.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T13:14:50.015 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T13:14:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 914315264 bytes 2024-09-16T13:14:50.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T13:14:50.018 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-16T13:14:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 980648960 bytes 2024-09-16T13:14:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 986970112 bytes 2024-09-16T13:14:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1083836416 bytes 2024-09-16T13:14:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1124836352 bytes 2024-09-16T13:14:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1170593792 bytes 2024-09-16T13:14:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1236519936 bytes 2024-09-16T13:14:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1253879808 bytes 2024-09-16T13:14:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1287554048 bytes 2024-09-16T13:14:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1360004096 bytes 2024-09-16T13:14:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1444519936 bytes 2024-09-16T13:14:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1487594496 bytes 2024-09-16T13:14:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1508719616 bytes 2024-09-16T13:14:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1537325056 bytes 2024-09-16T13:14:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1600838656 bytes 2024-09-16T13:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1689218048 bytes 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:14:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:14:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129bda964202 2024-09-16T13:14:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:14:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:14:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 present 129bda964202 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129bda964202 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:14:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:14:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129bda964202 2024-09-16T13:14:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129bda964202 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129bda964202 -o 129bda964202 = 129bda964202 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-16T13:14:53.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:14:53.742+0000 7f7b87fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2513800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2024-09-16T13:14:53.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:14:53.742+0000 7f7b87fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2513800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2024-09-16T13:14:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:14:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T13:14:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:14:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:14:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:14:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T13:14:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:14:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:14:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:14:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T13:14:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T13:14:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:14:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T13:14:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:14:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:14:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:14:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:14:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-16T13:14:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-16T13:14:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:14:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T13:14:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:14:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:14:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-16T13:14:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T13:14:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T13:15:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:15:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:15:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:15:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-16T13:15:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:15:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:15:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T13:15:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:15:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 193d3af88046 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.193d3af88046 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:15:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:15:23 2024 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:15:23 2024 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:15:23 2024 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 49444fb5-347f-4111-821b-dfbcdc6b1f36 2024-09-16T13:15:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:15:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T13:15:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:15:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:15:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:15:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:15:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 15388672 bytes 2024-09-16T13:15:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 57202688 bytes 2024-09-16T13:15:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 148652032 bytes 2024-09-16T13:15:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 202150912 bytes 2024-09-16T13:15:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 246137856 bytes 2024-09-16T13:15:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 258546688 bytes 2024-09-16T13:15:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 304693248 bytes 2024-09-16T13:15:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 349964288 bytes 2024-09-16T13:15:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 427663360 bytes 2024-09-16T13:15:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 451118080 bytes 2024-09-16T13:15:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 482134016 bytes 2024-09-16T13:15:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 520112128 bytes 2024-09-16T13:15:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 598044672 bytes 2024-09-16T13:15:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 649527296 bytes 2024-09-16T13:15:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 718608384 bytes 2024-09-16T13:15:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 728975360 bytes 2024-09-16T13:15:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 747374592 bytes 2024-09-16T13:15:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 783860736 bytes 2024-09-16T13:15:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 828452864 bytes 2024-09-16T13:15:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 861826048 bytes 2024-09-16T13:15:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 879804416 bytes 2024-09-16T13:15:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 920809472 bytes 2024-09-16T13:15:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1008087040 bytes 2024-09-16T13:15:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1051419648 bytes 2024-09-16T13:15:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1122579456 bytes 2024-09-16T13:15:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1142425600 bytes 2024-09-16T13:15:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1147632640 bytes 2024-09-16T13:15:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1167870976 bytes 2024-09-16T13:15:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1192004608 bytes 2024-09-16T13:15:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1206874112 bytes 2024-09-16T13:15:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1229159424 bytes 2024-09-16T13:15:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1231834112 bytes 2024-09-16T13:15:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1318708224 bytes 2024-09-16T13:15:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1327105024 bytes 2024-09-16T13:15:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1346123776 bytes 2024-09-16T13:15:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1361073152 bytes 2024-09-16T13:15:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1398044672 bytes 2024-09-16T13:15:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1405778944 bytes 2024-09-16T13:15:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1519127552 bytes 2024-09-16T13:15:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1558474752 bytes 2024-09-16T13:15:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1564340224 bytes 2024-09-16T13:15:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1641935872 bytes 2024-09-16T13:15:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1700635648 bytes 2024-09-16T13:15:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1702634496 bytes 2024-09-16T13:15:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1821232128 bytes 2024-09-16T13:15:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1890191360 bytes 2024-09-16T13:15:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1922096128 bytes 2024-09-16T13:15:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1959015424 bytes 2024-09-16T13:15:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1997587456 bytes 2024-09-16T13:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2020413440 bytes 2024-09-16T13:15:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2053935104 bytes 2024-09-16T13:15:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2054566912 bytes 2024-09-16T13:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2084762624 bytes 2024-09-16T13:15:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2172298240 bytes 2024-09-16T13:15:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2219805696 bytes 2024-09-16T13:15:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2260260864 bytes 2024-09-16T13:15:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2292408320 bytes 2024-09-16T13:15:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2335745024 bytes 2024-09-16T13:15:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2390615040 bytes 2024-09-16T13:15:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2467990528 bytes 2024-09-16T13:15:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2488877056 bytes 2024-09-16T13:15:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2556299264 bytes 2024-09-16T13:15:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2595651584 bytes 2024-09-16T13:15:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2660681728 bytes 2024-09-16T13:15:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2716113920 bytes 2024-09-16T13:15:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2782901248 bytes 2024-09-16T13:15:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2801943552 bytes 2024-09-16T13:15:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2802568192 bytes 2024-09-16T13:15:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2817925120 bytes 2024-09-16T13:15:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2862731264 bytes 2024-09-16T13:15:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2872169472 bytes 2024-09-16T13:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2903100416 bytes 2024-09-16T13:15:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2943923200 bytes 2024-09-16T13:15:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2956961792 bytes 2024-09-16T13:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2989826048 bytes 2024-09-16T13:15:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:15:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:15:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:15:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129b6de80562 2024-09-16T13:15:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:15:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:15:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 present 129b6de80562 2024-09-16T13:15:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:15:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b6de80562 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:15:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b6de80562 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b6de80562 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b6de80562 -o 129b6de80562 = 129b6de80562 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:15:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:15:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-16T13:15:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-16T13:15:37.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:15:37.078+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2541400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2024-09-16T13:15:37.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:15:37.078+0000 7f7b877dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560fd2541400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2024-09-16T13:15:40.837 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7228150497491361) > weight (0.25) 2024-09-16T13:15:40.837 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T13:15:40.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T13:15:40.838 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T13:15:40.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:15:40.836+0000 7f90691f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49204) UID: 1000 2024-09-16T13:15:41.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:15:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:15:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:15:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:15:42.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:15:42.106+0000 7fae775f7640 -1 rbd::mirror::ImageMap: 0x5627a5418900 handle_peer_ack_remove: failed to remove global_image_id=49444fb5-347f-4111-821b-dfbcdc6b1f36 2024-09-16T13:15:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T13:15:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:15:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:15:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T13:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T13:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:15:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T13:15:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-16T13:15:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T13:15:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:15:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:15:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T13:15:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:15:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:15:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-16T13:15:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T13:15:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T13:15:49.839 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T13:15:49.839 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:15:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:15:49.840 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T13:15:49.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T13:15:49.840 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T13:15:49.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T13:15:49.844 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-16T13:16:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:16:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T13:16:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:16:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 195a38d388d9 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.195a38d388d9 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:16:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:16:05 2024 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:16:05 2024 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:16:05 2024 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ba48e699-db41-40b9-8dba-b8f1e60d26f2 2024-09-16T13:16:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:16:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T13:16:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:16:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-16T13:16:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:16:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:16:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 11308032 bytes 2024-09-16T13:16:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 132490240 bytes 2024-09-16T13:16:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 199400448 bytes 2024-09-16T13:16:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 254303232 bytes 2024-09-16T13:16:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 312358912 bytes 2024-09-16T13:16:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 334069760 bytes 2024-09-16T13:16:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 347368448 bytes 2024-09-16T13:16:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 356451328 bytes 2024-09-16T13:16:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 385529856 bytes 2024-09-16T13:16:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 466079744 bytes 2024-09-16T13:16:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 499489792 bytes 2024-09-16T13:16:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 522116096 bytes 2024-09-16T13:16:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 539022336 bytes 2024-09-16T13:16:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 603216896 bytes 2024-09-16T13:16:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 603845632 bytes 2024-09-16T13:16:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 625306624 bytes 2024-09-16T13:16:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 649188352 bytes 2024-09-16T13:16:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 658899968 bytes 2024-09-16T13:16:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 712526848 bytes 2024-09-16T13:16:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 779593728 bytes 2024-09-16T13:16:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 811583488 bytes 2024-09-16T13:16:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 817921024 bytes 2024-09-16T13:16:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 881216512 bytes 2024-09-16T13:16:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 927190016 bytes 2024-09-16T13:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 956769280 bytes 2024-09-16T13:16:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 957377536 bytes 2024-09-16T13:16:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 982538240 bytes 2024-09-16T13:16:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 998098944 bytes 2024-09-16T13:16:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1027524608 bytes 2024-09-16T13:16:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1102281728 bytes 2024-09-16T13:16:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1117869056 bytes 2024-09-16T13:16:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1125630976 bytes 2024-09-16T13:16:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1126876160 bytes 2024-09-16T13:16:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1225845760 bytes 2024-09-16T13:16:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1242901504 bytes 2024-09-16T13:16:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1277186048 bytes 2024-09-16T13:16:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1307604992 bytes 2024-09-16T13:16:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1316076544 bytes 2024-09-16T13:16:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1356392448 bytes 2024-09-16T13:16:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1420252160 bytes 2024-09-16T13:16:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1451501568 bytes 2024-09-16T13:16:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1503608832 bytes 2024-09-16T13:16:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1515526144 bytes 2024-09-16T13:16:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1565877248 bytes 2024-09-16T13:16:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1571663872 bytes 2024-09-16T13:16:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1581205504 bytes 2024-09-16T13:16:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1642025984 bytes 2024-09-16T13:16:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1726559232 bytes 2024-09-16T13:16:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1749498880 bytes 2024-09-16T13:16:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1769141248 bytes 2024-09-16T13:16:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1839765504 bytes 2024-09-16T13:16:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1884858368 bytes 2024-09-16T13:16:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1895528448 bytes 2024-09-16T13:16:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1931807744 bytes 2024-09-16T13:16:14.019 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9991432877062952) > weight (0.25) 2024-09-16T13:16:14.019 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.33852593900404415) > weight (0.25) 2024-09-16T13:16:14.019 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T13:16:14.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T13:16:14.019 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T13:16:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:16:14.018+0000 7f7b957f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 48932) UID: 1000 2024-09-16T13:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 1976497152 bytes 2024-09-16T13:16:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:16:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2029145088 bytes 2024-09-16T13:16:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2136594432 bytes 2024-09-16T13:16:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2144667648 bytes 2024-09-16T13:16:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2194548736 bytes 2024-09-16T13:16:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2204229632 bytes 2024-09-16T13:16:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2249327616 bytes 2024-09-16T13:16:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2289081344 bytes 2024-09-16T13:16:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2368758784 bytes 2024-09-16T13:16:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2379844608 bytes 2024-09-16T13:16:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2393433088 bytes 2024-09-16T13:16:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2394038272 bytes 2024-09-16T13:16:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2448970752 bytes 2024-09-16T13:16:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2502282240 bytes 2024-09-16T13:16:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2564535296 bytes 2024-09-16T13:16:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2591806464 bytes 2024-09-16T13:16:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2624112640 bytes 2024-09-16T13:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2674577408 bytes 2024-09-16T13:16:17.020 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T13:16:17.020 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:16:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:16:17.021 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T13:16:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T13:16:17.021 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T13:16:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T13:16:17.024 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-16T13:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2705853440 bytes 2024-09-16T13:16:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2714042368 bytes 2024-09-16T13:16:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2748024832 bytes 2024-09-16T13:16:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2778374144 bytes 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129b86a3a0af 2024-09-16T13:16:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:16:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 present 129b86a3a0af 2024-09-16T13:16:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-16T13:16:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:16:19.021+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a65fa4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:16:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:16:19.021+0000 7fae77df8640 -1 rbd::mirror::ImageReplayer: 0x5627a64ed680 [3/ba48e699-db41-40b9-8dba-b8f1e60d26f2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:16:23.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:16:23.793+0000 7f29985f8640 -1 rbd::mirror::ImageMap: 0x5599169f4120 handle_peer_ack_remove: failed to remove global_image_id=ba48e699-db41-40b9-8dba-b8f1e60d26f2 2024-09-16T13:16:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=129b86a3a0af 2024-09-16T13:16:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = present 2024-09-16T13:16:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = present 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-16T13:16:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = present 2024-09-16T13:16:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 4 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = present 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 8 2024-09-16T13:16:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=129b86a3a0af 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 129b86a3a0af 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 129b86a3a0af -o 129b86a3a0af = 129b86a3a0af 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = present 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 8 2024-09-16T13:16:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted 129b86a3a0af 2024-09-16T13:16:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=129b86a3a0af 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-16T13:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T13:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T13:16:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:16:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:16:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:16:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-16T13:16:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:16:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T13:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-16T13:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T13:16:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T13:16:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:16:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2024-09-16T13:16:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-16T13:16:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:16:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T13:16:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:16:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T13:16:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-16T13:16:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:16:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:16:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 196f45f8c411 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.196f45f8c411 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:16:47 2024 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:16:47 2024 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:16:47 2024 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c124a830-3aa5-4e4e-8672-4598cc2d6bea 2024-09-16T13:16:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:16:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T13:16:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:16:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:16:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:16:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:16:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 2034688 bytes 2024-09-16T13:16:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 48717824 bytes 2024-09-16T13:16:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 119427072 bytes 2024-09-16T13:16:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 154737664 bytes 2024-09-16T13:16:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 177806336 bytes 2024-09-16T13:16:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 268911616 bytes 2024-09-16T13:16:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 308112384 bytes 2024-09-16T13:16:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 359421952 bytes 2024-09-16T13:16:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 402931712 bytes 2024-09-16T13:16:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 522503168 bytes 2024-09-16T13:16:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 526163968 bytes 2024-09-16T13:16:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 591830016 bytes 2024-09-16T13:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 635458560 bytes 2024-09-16T13:16:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 722340864 bytes 2024-09-16T13:16:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 727691264 bytes 2024-09-16T13:16:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 760601600 bytes 2024-09-16T13:16:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 876007424 bytes 2024-09-16T13:16:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 925447168 bytes 2024-09-16T13:16:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 941533184 bytes 2024-09-16T13:16:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 960918528 bytes 2024-09-16T13:16:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 985291776 bytes 2024-09-16T13:16:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 991766528 bytes 2024-09-16T13:16:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1033287680 bytes 2024-09-16T13:16:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1161308160 bytes 2024-09-16T13:16:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1181239296 bytes 2024-09-16T13:16:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1253198848 bytes 2024-09-16T13:16:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1343801344 bytes 2024-09-16T13:16:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1377433600 bytes 2024-09-16T13:16:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1423158272 bytes 2024-09-16T13:16:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1450038272 bytes 2024-09-16T13:16:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1497570304 bytes 2024-09-16T13:16:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1541134336 bytes 2024-09-16T13:16:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1572017152 bytes 2024-09-16T13:16:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1621305344 bytes 2024-09-16T13:16:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:16:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:16:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:16:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:16:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=165478c7bf32 2024-09-16T13:16:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:16:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:16:54.844 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.822699346128729) > weight (0.25) 2024-09-16T13:16:54.845 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.26338306038511117) > weight (0.25) 2024-09-16T13:16:54.845 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7272760498241002) > weight (0.25) 2024-09-16T13:16:54.845 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8353221587683102) > weight (0.25) 2024-09-16T13:16:54.845 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T13:16:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 present 165478c7bf32 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=165478c7bf32 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:16:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:16:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=165478c7bf32 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 165478c7bf32 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 165478c7bf32 -o 165478c7bf32 = 165478c7bf32 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:16:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-16T13:16:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:16:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:16:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:16:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-16T13:16:55.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:16:55.870+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55991918f000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2024-09-16T13:16:55.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:16:55.870+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55991918f000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2024-09-16T13:17:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:17:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:17:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:17:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:17:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:00.899 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T13:17:00.897+0000 7f6068a65640 -1 reset not still connected to 0x55b6ed50d520 2024-09-16T13:17:00.899 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T13:17:00.897+0000 7f6068a65640 -1 reset not still connected to 0x55b6ee47ac30 2024-09-16T13:17:00.899 INFO:tasks.ceph.cluster1.osd.2.smithi177.stderr:2024-09-16T13:17:00.897+0000 7f6068a65640 -1 reset not still connected to 0x55b6f1ad1110 2024-09-16T13:17:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T13:17:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:17:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:17:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:17:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T13:17:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:17:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-16T13:17:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T13:17:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:17:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-16T13:17:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T13:17:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T13:17:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:17:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:17:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:17:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-16T13:17:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:17:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:17:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:17:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:17:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:17:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:17:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T13:17:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:17:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T13:17:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-16T13:17:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:17:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:17:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:17:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1981f1eed086 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1981f1eed086 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:17:25 2024 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:17:25 2024 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:17:25 2024 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 71051d3a-4309-4033-9e3d-c152926f6c47 2024-09-16T13:17:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T13:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:17:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-16T13:17:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:17:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:17:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 66400256 bytes 2024-09-16T13:17:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 119559168 bytes 2024-09-16T13:17:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 174033920 bytes 2024-09-16T13:17:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 249450496 bytes 2024-09-16T13:17:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 314174464 bytes 2024-09-16T13:17:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 316615680 bytes 2024-09-16T13:17:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 345705472 bytes 2024-09-16T13:17:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 365925376 bytes 2024-09-16T13:17:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 454803456 bytes 2024-09-16T13:17:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 518124544 bytes 2024-09-16T13:17:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 590258176 bytes 2024-09-16T13:17:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 622437376 bytes 2024-09-16T13:17:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 641134592 bytes 2024-09-16T13:17:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 648698880 bytes 2024-09-16T13:17:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 709391360 bytes 2024-09-16T13:17:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 755262464 bytes 2024-09-16T13:17:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 811418624 bytes 2024-09-16T13:17:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 832636928 bytes 2024-09-16T13:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 891658240 bytes 2024-09-16T13:17:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 920401920 bytes 2024-09-16T13:17:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 991435776 bytes 2024-09-16T13:17:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1034494976 bytes 2024-09-16T13:17:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1055918080 bytes 2024-09-16T13:17:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1079378944 bytes 2024-09-16T13:17:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1102038016 bytes 2024-09-16T13:17:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1128878080 bytes 2024-09-16T13:17:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1174670336 bytes 2024-09-16T13:17:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1187369984 bytes 2024-09-16T13:17:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1252826112 bytes 2024-09-16T13:17:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1332168704 bytes 2024-09-16T13:17:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1393011712 bytes 2024-09-16T13:17:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1447588864 bytes 2024-09-16T13:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1463983104 bytes 2024-09-16T13:17:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1479230464 bytes 2024-09-16T13:17:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1502496768 bytes 2024-09-16T13:17:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1539776512 bytes 2024-09-16T13:17:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1566740480 bytes 2024-09-16T13:17:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:17:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:17:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:17:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:17:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=1654b256a01a 2024-09-16T13:17:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:17:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 present 1654b256a01a 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=1654b256a01a 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:17:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:17:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=1654b256a01a 2024-09-16T13:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1654b256a01a 2024-09-16T13:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 1654b256a01a -o 1654b256a01a = 1654b256a01a 2024-09-16T13:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-16T13:17:32.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:17:32.778+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559919056000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2024-09-16T13:17:32.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:17:32.778+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559919056000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2024-09-16T13:17:34.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5924225176100173) > weight (0.25) 2024-09-16T13:17:34.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6262815258858297) > weight (0.25) 2024-09-16T13:17:34.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8894350779412613) > weight (0.25) 2024-09-16T13:17:34.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.338465173149871) > weight (0.25) 2024-09-16T13:17:34.026 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-16T13:17:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:17:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T13:17:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:17:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T13:17:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:17:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:17:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:17:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T13:17:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:17:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T13:17:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:17:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T13:17:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:17:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-16T13:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T13:17:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:17:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:17:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-16T13:17:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T13:17:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T13:18:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2024-09-16T13:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-16T13:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T13:18:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:18:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T13:18:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-16T13:18:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:18:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:18:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 199170ea24ed 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.199170ea24ed 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:18:02 2024 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:18:02 2024 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:18:02 2024 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ea9a1163-8ad2-4907-9614-4272f8f4c7c8 2024-09-16T13:18:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:18:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T13:18:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-16T13:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:18:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 46833664 bytes 2024-09-16T13:18:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 71783424 bytes 2024-09-16T13:18:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 139938816 bytes 2024-09-16T13:18:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 169555968 bytes 2024-09-16T13:18:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 219794432 bytes 2024-09-16T13:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 241476608 bytes 2024-09-16T13:18:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 279406592 bytes 2024-09-16T13:18:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 305337344 bytes 2024-09-16T13:18:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 311513088 bytes 2024-09-16T13:18:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 341959680 bytes 2024-09-16T13:18:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 376109056 bytes 2024-09-16T13:18:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 408783872 bytes 2024-09-16T13:18:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 417166336 bytes 2024-09-16T13:18:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 437178368 bytes 2024-09-16T13:18:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 439243776 bytes 2024-09-16T13:18:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 476231680 bytes 2024-09-16T13:18:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 555581440 bytes 2024-09-16T13:18:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 585076736 bytes 2024-09-16T13:18:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 648898560 bytes 2024-09-16T13:18:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 651978752 bytes 2024-09-16T13:18:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 666238976 bytes 2024-09-16T13:18:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 726619136 bytes 2024-09-16T13:18:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 747103232 bytes 2024-09-16T13:18:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 778820608 bytes 2024-09-16T13:18:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 816466944 bytes 2024-09-16T13:18:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 842415104 bytes 2024-09-16T13:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 912990208 bytes 2024-09-16T13:18:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 978857984 bytes 2024-09-16T13:18:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 979475456 bytes 2024-09-16T13:18:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1026048000 bytes 2024-09-16T13:18:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1077179392 bytes 2024-09-16T13:18:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1117977600 bytes 2024-09-16T13:18:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1236192256 bytes 2024-09-16T13:18:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1269163008 bytes 2024-09-16T13:18:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1277256704 bytes 2024-09-16T13:18:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1322339328 bytes 2024-09-16T13:18:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1398228992 bytes 2024-09-16T13:18:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1446064128 bytes 2024-09-16T13:18:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1479153664 bytes 2024-09-16T13:18:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1510193152 bytes 2024-09-16T13:18:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1561931776 bytes 2024-09-16T13:18:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1627101184 bytes 2024-09-16T13:18:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1665481728 bytes 2024-09-16T13:18:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1700392960 bytes 2024-09-16T13:18:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1748195328 bytes 2024-09-16T13:18:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1776049152 bytes 2024-09-16T13:18:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1828814848 bytes 2024-09-16T13:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1856244736 bytes 2024-09-16T13:18:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1861163008 bytes 2024-09-16T13:18:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1898759168 bytes 2024-09-16T13:18:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1925463040 bytes 2024-09-16T13:18:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1933887488 bytes 2024-09-16T13:18:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2018456576 bytes 2024-09-16T13:18:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2063888384 bytes 2024-09-16T13:18:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2103599104 bytes 2024-09-16T13:18:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2133571584 bytes 2024-09-16T13:18:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2202679296 bytes 2024-09-16T13:18:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2255131648 bytes 2024-09-16T13:18:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2424456192 bytes 2024-09-16T13:18:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2469318656 bytes 2024-09-16T13:18:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2538988544 bytes 2024-09-16T13:18:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2607941632 bytes 2024-09-16T13:18:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2653126656 bytes 2024-09-16T13:18:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2695349248 bytes 2024-09-16T13:18:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2733811712 bytes 2024-09-16T13:18:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2837758976 bytes 2024-09-16T13:18:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2844127232 bytes 2024-09-16T13:18:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2868015104 bytes 2024-09-16T13:18:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2909602816 bytes 2024-09-16T13:18:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2962268160 bytes 2024-09-16T13:18:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3002458112 bytes 2024-09-16T13:18:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3013931008 bytes 2024-09-16T13:18:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3068831744 bytes 2024-09-16T13:18:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3109494784 bytes 2024-09-16T13:18:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3164564480 bytes 2024-09-16T13:18:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3256914944 bytes 2024-09-16T13:18:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3288233984 bytes 2024-09-16T13:18:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3288841216 bytes 2024-09-16T13:18:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3352990720 bytes 2024-09-16T13:18:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3431962624 bytes 2024-09-16T13:18:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3534451712 bytes 2024-09-16T13:18:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3555225600 bytes 2024-09-16T13:18:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3621728256 bytes 2024-09-16T13:18:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:18:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:18:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:18:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:18:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=16544a947bf1 2024-09-16T13:18:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:18:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:18:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 present 16544a947bf1 2024-09-16T13:18:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=16544a947bf1 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:18:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=16544a947bf1 2024-09-16T13:18:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 16544a947bf1 2024-09-16T13:18:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 16544a947bf1 -o 16544a947bf1 = 16544a947bf1 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-16T13:18:18.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:18:18.097+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5599191e8000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2024-09-16T13:18:18.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:18:18.097+0000 7f298addd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5599191e8000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=388, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2024-09-16T13:18:19.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:18:19.022+0000 7fae715eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a654f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:18:19.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:18:19.022+0000 7fae775f7640 -1 rbd::mirror::ImageReplayer: 0x5627a64ed680 [3/ea9a1163-8ad2-4907-9614-4272f8f4c7c8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:18:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:18:19.040+0000 7fae70dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5627a654fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T13:18:19.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:18:19.040+0000 7fae77df8640 -1 rbd::mirror::ImageReplayer: 0x5627a64ed680 [3/ea9a1163-8ad2-4907-9614-4272f8f4c7c8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T13:18:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:18:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T13:18:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:18:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:18:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:18:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T13:18:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-16T13:18:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T13:18:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-16T13:18:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:18:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:18:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T13:18:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-16T13:18:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T13:18:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T13:18:43.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7329331693375002) > weight (0.25) 2024-09-16T13:18:43.846 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T13:18:43.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T13:18:43.847 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T13:18:43.847 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T13:18:43.847 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T13:18:43.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:18:43.846+0000 7f89bcbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 73339) UID: 1000 2024-09-16T13:18:44.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:18:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-16T13:18:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-16T13:18:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T13:18:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-16T13:18:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 19a31ed89874 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.19a31ed89874 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-16T13:18:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Mon Sep 16 13:18:47 2024 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Mon Sep 16 13:18:47 2024 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Mon Sep 16 13:18:47 2024 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c8df3aa5-af94-4c65-ae61-4a3bda1c5b5f 2024-09-16T13:18:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-16T13:18:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T13:18:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T13:18:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-16T13:18:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:18:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T13:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 29723648 bytes 2024-09-16T13:18:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 67174400 bytes 2024-09-16T13:18:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 98097152 bytes 2024-09-16T13:18:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 144626688 bytes 2024-09-16T13:18:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 251713536 bytes 2024-09-16T13:18:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 284750848 bytes 2024-09-16T13:18:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 296334336 bytes 2024-09-16T13:18:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 360764416 bytes 2024-09-16T13:18:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 377492480 bytes 2024-09-16T13:18:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 428607488 bytes 2024-09-16T13:18:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 466710528 bytes 2024-09-16T13:18:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 568169472 bytes 2024-09-16T13:18:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:18:49.950 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T13:18:49.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T13:18:49.950 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T13:18:49.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T13:18:49.953 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-16T13:18:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 568798208 bytes 2024-09-16T13:18:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 594577408 bytes 2024-09-16T13:18:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 625687552 bytes 2024-09-16T13:18:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 669946880 bytes 2024-09-16T13:18:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 705809408 bytes 2024-09-16T13:18:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 744969216 bytes 2024-09-16T13:18:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 767841280 bytes 2024-09-16T13:18:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 798523392 bytes 2024-09-16T13:18:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 799563776 bytes 2024-09-16T13:18:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 854128640 bytes 2024-09-16T13:18:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 948773888 bytes 2024-09-16T13:18:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 986645504 bytes 2024-09-16T13:18:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1077299200 bytes 2024-09-16T13:18:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1098938368 bytes 2024-09-16T13:18:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1134047232 bytes 2024-09-16T13:18:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1257624576 bytes 2024-09-16T13:18:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1282709504 bytes 2024-09-16T13:18:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1302107136 bytes 2024-09-16T13:18:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1302733824 bytes 2024-09-16T13:18:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1356622848 bytes 2024-09-16T13:18:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1378913280 bytes 2024-09-16T13:18:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1407141888 bytes 2024-09-16T13:18:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1439803392 bytes 2024-09-16T13:18:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1508714496 bytes 2024-09-16T13:18:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1520147456 bytes 2024-09-16T13:18:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1540380672 bytes 2024-09-16T13:18:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1574637568 bytes 2024-09-16T13:18:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1619294208 bytes 2024-09-16T13:18:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1671360512 bytes 2024-09-16T13:18:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1698689024 bytes 2024-09-16T13:18:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-16T13:18:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:18:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T13:18:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:18:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:18:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:18:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=1654e3ccad0a 2024-09-16T13:18:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:18:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:18:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 present 1654e3ccad0a 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=1654e3ccad0a 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:18:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:18:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:18:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=1654e3ccad0a 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1654e3ccad0a 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 1654e3ccad0a -o 1654e3ccad0a = 1654e3ccad0a 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:18:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-16T13:18:55.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:18:55.824+0000 7f298b5de640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559919007000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2024-09-16T13:18:55.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:18:55.824+0000 7f298b5de640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559919007000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=390, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=68392c8a-786e-4db9-983c-eb2f365ca1ca, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2024-09-16T13:19:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:19:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:19:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:19:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T13:19:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-16T13:19:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T13:19:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-16T13:19:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T13:19:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T13:19:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T13:19:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:19:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-16T13:19:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T13:19:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:19:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T13:19:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T13:19:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-16T13:19:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T13:19:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T13:19:15.026 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.40241759488261564) > weight (0.25) 2024-09-16T13:19:15.027 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.49661475132115995) > weight (0.25) 2024-09-16T13:19:15.027 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5196025778899342) > weight (0.25) 2024-09-16T13:19:15.027 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5851562201201558) > weight (0.25) 2024-09-16T13:19:15.027 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-16T13:19:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:19:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-16T13:19:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.cccYcKFUz2/rbd-mirror.test.log 2024-09-16T13:19:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-16 13:19:24 TEST: check if removed 'images'\''' OMAP are removed 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-16 13:19:24 TEST: check if removed images' OMAP are removed 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=status_global 2024-09-16T13:19:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=status_global 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c status_global 2024-09-16T13:19:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_ 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_ 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T13:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_ 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_map 2024-09-16T13:19:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-16T13:19:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T13:19:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=status_global 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-16T13:19:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=status_global 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c status_global 2024-09-16T13:19:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T13:19:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_ 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_ 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_ 2024-09-16T13:19:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T13:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_map 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-16T13:19:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-16T13:19:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T13:19:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-16T13:19:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-16T13:19:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cleanup 0 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code=0 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z '' ']' 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster instance 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T13:19:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T13:19:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' does not exist 2024-09-16T13:19:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T13:19:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T13:19:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' does not exist 2024-09-16T13:19:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T13:19:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T13:19:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' does not exist 2024-09-16T13:19:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-16T13:19:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T13:19:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' does not exist 2024-09-16T13:19:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T13:19:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirrors cluster1 2024-09-16T13:19:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-16T13:19:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:0 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:1 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:2 '' 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:2 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:3 '' 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:3 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirrors cluster2 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T13:19:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:0 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:1 '' 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:1 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:2 '' 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:2 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:3 '' 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:3 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z 1 ']' 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test '' = /tmp/tmp.cccYcKFUz2 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -Rf /tmp/tmp.cccYcKFUz2 2024-09-16T13:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:19:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo OK 2024-09-16T13:19:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ exit 0 2024-09-16T13:19:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:OK 2024-09-16T13:19:28.920 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T13:19:28.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-16T13:19:28.987 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-16T13:19:28.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.067+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x5599175f4600 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f29915ea640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5599193d4480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f2991deb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5599192666c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f2991deb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5599176b0480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f29915ea640 -1 rbd::mirror::PoolWatcher: 0x5599193d34a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f2991deb640 -1 rbd::mirror::PoolWatcher: 0x5599191db4a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f29915ea640 -1 rbd::mirror::PoolWatcher: 0x559918f91c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f298addd640 -1 rbd::mirror::PoolWatcher: 0x559918f91a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f298addd640 -1 rbd::mirror::PoolWatcher: 0x5599193983c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f298addd640 -1 rbd::mirror::PoolWatcher: 0x559919399860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x5599175f4600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T13:19:29.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.068+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x5599175f4600 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T13:19:29.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.137+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x55991831f800 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T13:19:29.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.138+0000 7f2983dcf640 -1 rbd::mirror::PoolWatcher: 0x5599193d30e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.138+0000 7f297ddc3640 -1 rbd::mirror::PoolWatcher: 0x559919399a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.138+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x55991831f800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T13:19:29.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.138+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x55991831f800 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T13:19:29.293 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:29.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:19:29.357 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:19:29.358 DEBUG:teuthology.orchestra.run.smithi191:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T13:19:29.422 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:19:29.422 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T13:19:29.432 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T13:19:29.433 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-09-16T13:19:29.433 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T13:19:29.433 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T13:19:29.441 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T13:19:29.442 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 14 2024-09-16T13:19:29.442 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T13:19:29.442 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.451 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-16T13:19:29.451 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.461+0000 7fae785f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 51208) UID: 1000 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.461+0000 7fae715eb640 -1 rbd::mirror::PoolWatcher: 0x5627a61f4780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.461+0000 7fae715eb640 -1 rbd::mirror::PoolWatcher: 0x5627a61f5e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.461+0000 7fae77df8640 -1 rbd::mirror::LeaderWatcher: 0x5627a5417800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae6adde640 -1 rbd::mirror::PoolWatcher: 0x5627a64d2b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae6adde640 -1 rbd::mirror::PoolWatcher: 0x5627a64d2d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae77df8640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5281 object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae775f7640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5279 object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae775f7640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5279 object: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae635cf640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae70dea640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.462+0000 7fae715eb640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae6adde640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5264360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae6adde640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5264480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae77df8640 -1 rbd::mirror::NamespaceReplayer: 0x5627a5262000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae77df8640 -1 rbd::mirror::NamespaceReplayer: 0x5627a52621a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae70dea640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5264000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae70dea640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5264120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae775f7640 -1 rbd::mirror::NamespaceReplayer: 0x5627a5262000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.463+0000 7fae775f7640 -1 rbd::mirror::NamespaceReplayer: 0x5627a52621a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.497+0000 7fae5cdc2640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5418a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.497+0000 7fae77df8640 -1 rbd::mirror::NamespaceReplayer: 0x5627a5f524e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.497+0000 7fae635cf640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a5385c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.497+0000 7fae775f7640 -1 rbd::mirror::NamespaceReplayer: 0x5627a5f524e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.499+0000 7fae70dea640 -1 rbd::mirror::PoolWatcher: 0x5627a61f4000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.499+0000 7fae6adde640 -1 rbd::mirror::PoolWatcher: 0x5627a6539e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.500+0000 7fae77df8640 -1 rbd::mirror::LeaderWatcher: 0x5627a46a2600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.500+0000 7fae77df8640 -1 rbd::mirror::LeaderWatcher: 0x5627a46a2600 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.500+0000 7fae775f7640 -1 rbd::mirror::LeaderWatcher: 0x5627a46a2600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.500+0000 7fae775f7640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5279 object: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.500+0000 7fae715eb640 -1 rbd::mirror::InstanceWatcher: 0x5627a20a3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.501+0000 7fae6a5dd640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a4750240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.501+0000 7fae775f7640 -1 rbd::mirror::NamespaceReplayer: 0x5627a46a04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.501+0000 7fae70dea640 -1 rbd::mirror::MirrorStatusUpdater 0x5627a3aed560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.501+0000 7fae77df8640 -1 rbd::mirror::NamespaceReplayer: 0x5627a46a04e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.516 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3' 2024-09-16T13:19:29.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-16T13:19:29.517 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.527 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-16T13:19:29.527 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.555+0000 7f51ae3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 72265) UID: 1000 2024-09-16T13:19:29.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::LeaderWatcher: 0x55ac6b9ef800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6404 object: (2) No such file or directory 2024-09-16T13:19:29.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6404 object: (2) No such file or directory 2024-09-16T13:19:29.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51a73eb640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6407 object: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51a73eb640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51993cf640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51993cf640 -1 rbd::mirror::MirrorStatusUpdater 0x55ac6b9b5e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::NamespaceReplayer: 0x55ac6c52e4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51a0bde640 -1 rbd::mirror::MirrorStatusUpdater 0x55ac6872dc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51a0bde640 -1 rbd::mirror::MirrorStatusUpdater 0x55ac6b850360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::NamespaceReplayer: 0x55ac6b84e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.556+0000 7f51adbf8640 -1 rbd::mirror::NamespaceReplayer: 0x55ac6b84e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.558+0000 7f51adbf8640 -1 rbd::mirror::LeaderWatcher: 0x55ac6acb4600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.558+0000 7f51adbf8640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6404 object: (2) No such file or directory 2024-09-16T13:19:29.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.559+0000 7f51a6bea640 -1 rbd::mirror::InstanceWatcher: 0x55ac686d7180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.559+0000 7f51a03dd640 -1 rbd::mirror::MirrorStatusUpdater 0x55ac6ad58240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.559+0000 7f51adbf8640 -1 rbd::mirror::NamespaceReplayer: 0x55ac6aca04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.559+0000 7f51a6bea640 -1 rbd::mirror::MirrorStatusUpdater 0x55ac6a103440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.559+0000 7f51ad3f7640 -1 rbd::mirror::NamespaceReplayer: 0x55ac6aca04e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.573 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2' 2024-09-16T13:19:29.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-16T13:19:29.574 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.583 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-16T13:19:29.583 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.612+0000 7f6b311f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 74946) UID: 1000 2024-09-16T13:19:29.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b301f7640 -1 rbd::mirror::LeaderWatcher: 0x562a79d3f800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b309f8640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6579 object: (2) No such file or directory 2024-09-16T13:19:29.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b301f7640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6579 object: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b2a1eb640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b301f7640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6582 object: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b299ea640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b1c1cf640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b231dd640 -1 rbd::mirror::MirrorStatusUpdater 0x562a76a29c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b309f8640 -1 rbd::mirror::NamespaceReplayer: 0x562a79b9e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b239de640 -1 rbd::mirror::MirrorStatusUpdater 0x562a79ba0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.613+0000 7f6b301f7640 -1 rbd::mirror::NamespaceReplayer: 0x562a79b9e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b159c2640 -1 rbd::mirror::MirrorStatusUpdater 0x562a7a8d25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b309f8640 -1 rbd::mirror::NamespaceReplayer: 0x562a7a8d0340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b299ea640 -1 rbd::mirror::MirrorStatusUpdater 0x562a79ba0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b301f7640 -1 rbd::mirror::NamespaceReplayer: 0x562a79b9e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b2a1eb640 -1 rbd::mirror::MirrorStatusUpdater 0x562a79ba0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b309f8640 -1 rbd::mirror::NamespaceReplayer: 0x562a79b9e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b1c9d0640 -1 rbd::mirror::MirrorStatusUpdater 0x562a79d70000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.614+0000 7f6b301f7640 -1 rbd::mirror::NamespaceReplayer: 0x562a7a8d0340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.615+0000 7f6b309f8640 -1 rbd::mirror::LeaderWatcher: 0x562a78ff0600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.616+0000 7f6b309f8640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6579 object: (2) No such file or directory 2024-09-16T13:19:29.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.616+0000 7f6b299ea640 -1 rbd::mirror::InstanceWatcher: 0x562a769d3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.616+0000 7f6b231dd640 -1 rbd::mirror::MirrorStatusUpdater 0x562a790a0240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.616+0000 7f6b309f8640 -1 rbd::mirror::NamespaceReplayer: 0x562a78fe64e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.616+0000 7f6b299ea640 -1 rbd::mirror::MirrorStatusUpdater 0x562a7841f560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.617+0000 7f6b301f7640 -1 rbd::mirror::NamespaceReplayer: 0x562a78fe64e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.629 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2024-09-16T13:19:29.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-16T13:19:29.629 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.640 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-16T13:19:29.640 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.668+0000 7f48539f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63106) UID: 1000 2024-09-16T13:19:29.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.668+0000 7f483f1d0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558f2cb5c6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T13:19:29.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48531f8640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5832 object: (2) No such file or directory 2024-09-16T13:19:29.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f483f1d0640 -1 rbd::mirror::PoolWatcher: 0x558f2e5bb4a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48529f7640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5832 object: (2) No such file or directory 2024-09-16T13:19:29.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f484c9eb640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f484c9eb640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48381c2640 -1 rbd::mirror::PoolWatcher: 0x558f2e45fa40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T13:19:29.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48459dd640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2a505c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48459dd640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2cbac360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2cbaa000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2cbaa1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48529f7640 -1 rbd::mirror::LeaderWatcher: 0x558f2d7dd800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48529f7640 -1 rbd::mirror::LeaderWatcher: 0x558f2d7dd800 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f484c1ea640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2cbac000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f484c1ea640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2cbac120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2cbaa000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.669+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2cbaa1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.670+0000 7f48529f7640 -1 rbd::mirror::LeaderWatcher: 0x558f2d7dd800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.670+0000 7f48529f7640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4afc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5834 object: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.670+0000 7f483f1d0640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4afc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.671+0000 7f48381c2640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2e3345a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.671+0000 7f48529f7640 -1 rbd::mirror::NamespaceReplayer: 0x558f2e3224e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.671+0000 7f483f1d0640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2d783e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.671+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2e3224e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.673+0000 7f48529f7640 -1 rbd::mirror::LeaderWatcher: 0x558f2caa2600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.673+0000 7f48529f7640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5832 object: (2) No such file or directory 2024-09-16T13:19:29.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.673+0000 7f484c9eb640 -1 rbd::mirror::InstanceWatcher: 0x558f2a4af180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.675+0000 7f48461de640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2cb4c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.675+0000 7f48529f7640 -1 rbd::mirror::NamespaceReplayer: 0x558f2ca8e4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.675+0000 7f484c1ea640 -1 rbd::mirror::MirrorStatusUpdater 0x558f2bee3440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.675+0000 7f48531f8640 -1 rbd::mirror::NamespaceReplayer: 0x558f2ca8e4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.689 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2024-09-16T13:19:29.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-16T13:19:29.690 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.700 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-16T13:19:29.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.729+0000 7f2998df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 64903) UID: 1000 2024-09-16T13:19:29.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29985f8640 -1 rbd::mirror::LeaderWatcher: 0x55991831f800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29985f8640 -1 rbd::mirror::InstanceWatcher: 0x559915007500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5716 object: (2) No such file or directory 2024-09-16T13:19:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f2997df7640 -1 rbd::mirror::InstanceWatcher: 0x559918f40000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5716 object: (2) No such file or directory 2024-09-16T13:19:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f2997df7640 -1 rbd::mirror::InstanceWatcher: 0x559915007880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5719 object: (2) No such file or directory 2024-09-16T13:19:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29915ea640 -1 rbd::mirror::InstanceWatcher: 0x559915007500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29915ea640 -1 rbd::mirror::InstanceWatcher: 0x559918f40000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f2983dcf640 -1 rbd::mirror::InstanceWatcher: 0x559915007880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f298b5de640 -1 rbd::mirror::MirrorStatusUpdater 0x559917710360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f298b5de640 -1 rbd::mirror::MirrorStatusUpdater 0x55991505dc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f2997df7640 -1 rbd::mirror::NamespaceReplayer: 0x55991770e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f2997df7640 -1 rbd::mirror::NamespaceReplayer: 0x55991770e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29915ea640 -1 rbd::mirror::MirrorStatusUpdater 0x559917710000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.730+0000 7f29915ea640 -1 rbd::mirror::MirrorStatusUpdater 0x559917710120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f29985f8640 -1 rbd::mirror::NamespaceReplayer: 0x55991770e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f29985f8640 -1 rbd::mirror::NamespaceReplayer: 0x55991770e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f297d5c2640 -1 rbd::mirror::MirrorStatusUpdater 0x55991505de60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f2997df7640 -1 rbd::mirror::NamespaceReplayer: 0x55991770eb60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f29845d0640 -1 rbd::mirror::MirrorStatusUpdater 0x5599182c9d40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.731+0000 7f29985f8640 -1 rbd::mirror::NamespaceReplayer: 0x55991770eb60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.733+0000 7f2997df7640 -1 rbd::mirror::LeaderWatcher: 0x5599175f4600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.733+0000 7f2997df7640 -1 rbd::mirror::InstanceWatcher: 0x559915007180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5716 object: (2) No such file or directory 2024-09-16T13:19:29.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.733+0000 7f2991deb640 -1 rbd::mirror::InstanceWatcher: 0x559915007180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.734+0000 7f298addd640 -1 rbd::mirror::MirrorStatusUpdater 0x55991769c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.734+0000 7f2997df7640 -1 rbd::mirror::NamespaceReplayer: 0x5599175de4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.734+0000 7f29915ea640 -1 rbd::mirror::MirrorStatusUpdater 0x559916a45440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-16T13:19:29.734+0000 7f29985f8640 -1 rbd::mirror::NamespaceReplayer: 0x5599175de4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.750 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3' 2024-09-16T13:19:29.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-16T13:19:29.751 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.761 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-16T13:19:29.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.789+0000 7fc61bff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 73533) UID: 1000 2024-09-16T13:19:29.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61aff7640 -1 rbd::mirror::LeaderWatcher: 0x55f3a1687800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61b7f8640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6144 object: (2) No such file or directory 2024-09-16T13:19:29.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61b7f8640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6144 object: (2) No such file or directory 2024-09-16T13:19:29.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61b7f8640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6147 object: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc614feb640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc6147ea640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc606fcf640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc6147ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3a0a82000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc6147ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3a0a82120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61b7f8640 -1 rbd::mirror::NamespaceReplayer: 0x55f3a0a28820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.790+0000 7fc61b7f8640 -1 rbd::mirror::NamespaceReplayer: 0x55f3a0a289c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.791+0000 7fc6007c2640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3a16946c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.791+0000 7fc61aff7640 -1 rbd::mirror::NamespaceReplayer: 0x55f3a21dc4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.791+0000 7fc6077d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55f3a1605e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.791+0000 7fc61b7f8640 -1 rbd::mirror::NamespaceReplayer: 0x55f3a21dc4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.793+0000 7fc61aff7640 -1 rbd::mirror::LeaderWatcher: 0x55f3a0968600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.793+0000 7fc61aff7640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6144 object: (2) No such file or directory 2024-09-16T13:19:29.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.793+0000 7fc6147ea640 -1 rbd::mirror::InstanceWatcher: 0x55f39e355180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.794+0000 7fc614feb640 -1 rbd::mirror::MirrorStatusUpdater 0x55f39fe1b560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-16T13:19:29.794+0000 7fc61aff7640 -1 rbd::mirror::NamespaceReplayer: 0x55f3a0a28000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.806 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2' 2024-09-16T13:19:29.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-16T13:19:29.806 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.817 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-16T13:19:29.817 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.845+0000 7f83459f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 67595) UID: 1000 2024-09-16T13:19:29.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83451f8640 -1 rbd::mirror::LeaderWatcher: 0x56111893f800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83451f8640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5741 object: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83449f7640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5741 object: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83451f8640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5744 object: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f833e9eb640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f833e1ea640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f833e9eb640 -1 rbd::mirror::MirrorStatusUpdater 0x56111878c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83449f7640 -1 rbd::mirror::NamespaceReplayer: 0x56111877e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83309cf640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f833e9eb640 -1 rbd::mirror::MirrorStatusUpdater 0x56111878c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.846+0000 7f83451f8640 -1 rbd::mirror::NamespaceReplayer: 0x56111877e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.847+0000 7f832a1c2640 -1 rbd::mirror::MirrorStatusUpdater 0x56111894e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.847+0000 7f83449f7640 -1 rbd::mirror::NamespaceReplayer: 0x56111948c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.847+0000 7f83311d0640 -1 rbd::mirror::MirrorStatusUpdater 0x56111894c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.847+0000 7f83451f8640 -1 rbd::mirror::NamespaceReplayer: 0x56111948c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.849+0000 7f83449f7640 -1 rbd::mirror::LeaderWatcher: 0x561117c02600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.849+0000 7f83449f7640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5741 object: (2) No such file or directory 2024-09-16T13:19:29.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.850+0000 7f833e9eb640 -1 rbd::mirror::InstanceWatcher: 0x5611155f9180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.850+0000 7f83379dd640 -1 rbd::mirror::MirrorStatusUpdater 0x561117ca0240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.850+0000 7f83449f7640 -1 rbd::mirror::NamespaceReplayer: 0x561117bec4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.850+0000 7f833e1ea640 -1 rbd::mirror::MirrorStatusUpdater 0x561117041560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-16T13:19:29.850+0000 7f83451f8640 -1 rbd::mirror::NamespaceReplayer: 0x561117bec4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.863 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1' 2024-09-16T13:19:29.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-16T13:19:29.864 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T13:19:29.874 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-16T13:19:29.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:29.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.902+0000 7fb5399f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72810) UID: 1000 2024-09-16T13:19:29.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5391f8640 -1 rbd::mirror::LeaderWatcher: 0x55c2b1247800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5391f8640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6107 object: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5391f8640 -1 rbd::mirror::InstanceWatcher: 0x55c2b13ec000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6107 object: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5391f8640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6110 object: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5329eb640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5329eb640 -1 rbd::mirror::InstanceWatcher: 0x55c2b13ec000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5251d0640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb52b9dd640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2ad4cfc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5389f7640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b061c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb52b9dd640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2b061e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.903+0000 7fb5389f7640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b061c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5329eb640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2b061e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5329eb640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2b061e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5391f8640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b061c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5391f8640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b061c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb51e9c3640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2ad4cfe60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5389f7640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b12fe4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5249cf640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2b0733b00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.904+0000 7fb5391f8640 -1 rbd::mirror::NamespaceReplayer: 0x55c2b12fe4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.906+0000 7fb5389f7640 -1 rbd::mirror::LeaderWatcher: 0x55c2afa8a600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T13:19:29.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.906+0000 7fb5389f7640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6107 object: (2) No such file or directory 2024-09-16T13:19:29.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.906+0000 7fb5329eb640 -1 rbd::mirror::InstanceWatcher: 0x55c2ad479180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T13:19:29.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.907+0000 7fb52c1de640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2aeec9e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.907+0000 7fb5389f7640 -1 rbd::mirror::NamespaceReplayer: 0x55c2afa704e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.907+0000 7fb5329eb640 -1 rbd::mirror::MirrorStatusUpdater 0x55c2aeec9560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T13:19:29.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-16T13:19:29.907+0000 7fb5391f8640 -1 rbd::mirror::NamespaceReplayer: 0x55c2afa704e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T13:19:29.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:daemon-helper: command failed with exit status 1 2024-09-16T13:19:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:19:29.920 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0' 2024-09-16T13:19:29.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-16T13:19:29.920 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:19:29.931 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-09-16T13:19:29.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T13:19:30.222 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:30.222 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":455,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T13:19:28.731527+0000","last_up_change":"2024-09-16T12:43:07.180410+0000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:43:07.438466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:43:13.318047+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":33180},{"type":"v1","addr":"172.21.15.191:6801","nonce":33180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":33180},{"type":"v1","addr":"172.21.15.191:6803","nonce":33180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":33180},{"type":"v1","addr":"172.21.15.191:6807","nonce":33180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":33180},{"type":"v1","addr":"172.21.15.191:6805","nonce":33180}]},"public_addr":"172.21.15.191:6801/33180","cluster_addr":"172.21.15.191:6803/33180","heartbeat_back_addr":"172.21.15.191:6807/33180","heartbeat_front_addr":"172.21.15.191:6805/33180","state":["exists","up"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":33181},{"type":"v1","addr":"172.21.15.191:6812","nonce":33181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":33181},{"type":"v1","addr":"172.21.15.191:6816","nonce":33181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":33181},{"type":"v1","addr":"172.21.15.191:6823","nonce":33181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":33181},{"type":"v1","addr":"172.21.15.191:6820","nonce":33181}]},"public_addr":"172.21.15.191:6812/33181","cluster_addr":"172.21.15.191:6816/33181","heartbeat_back_addr":"172.21.15.191:6823/33181","heartbeat_front_addr":"172.21.15.191:6820/33181","state":["exists","up"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":33182},{"type":"v1","addr":"172.21.15.191:6809","nonce":33182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6811","nonce":33182},{"type":"v1","addr":"172.21.15.191:6813","nonce":33182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":33182},{"type":"v1","addr":"172.21.15.191:6821","nonce":33182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":33182},{"type":"v1","addr":"172.21.15.191:6817","nonce":33182}]},"public_addr":"172.21.15.191:6809/33182","cluster_addr":"172.21.15.191:6813/33182","heartbeat_back_addr":"172.21.15.191:6821/33182","heartbeat_front_addr":"172.21.15.191:6817/33182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.365008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.385578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.340888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:0/4013810156":"2024-09-16T14:18:00.816387+0000","172.21.15.191:0/1193464417":"2024-09-16T14:17:23.902409+0000","172.21.15.191:0/3070994826":"2024-09-16T14:16:47.337609+0000","172.21.15.191:0/910748197":"2024-09-16T14:16:03.861334+0000","172.21.15.191:0/1238304107":"2024-09-16T14:14:07.778628+0000","172.21.15.191:0/1862466608":"2024-09-16T14:10:13.658094+0000","172.21.15.191:0/2594442942":"2024-09-16T14:09:01.613262+0000","172.21.15.191:0/2765424310":"2024-09-16T14:05:59.722885+0000","172.21.15.191:0/1455260247":"2024-09-16T14:19:24.264655+0000","172.21.15.191:0/1945055819":"2024-09-16T14:13:25.678957+0000","172.21.15.191:0/4266714643":"2024-09-16T13:45:39.236851+0000","172.21.15.191:0/3084174430":"2024-09-16T14:08:10.999696+0000","172.21.15.191:0/2189733109":"2024-09-16T14:05:51.704200+0000","172.21.15.191:0/1630258868":"2024-09-16T14:05:26.538888+0000","172.21.15.191:0/4255407941":"2024-09-16T13:50:47.225084+0000","172.21.15.191:0/2639548626":"2024-09-16T14:05:18.501394+0000","172.21.15.191:0/2705217819":"2024-09-16T14:04:44.153373+0000","172.21.15.191:0/3654451968":"2024-09-16T13:47:12.403445+0000","172.21.15.191:0/3433548665":"2024-09-16T14:08:41.036573+0000","172.21.15.191:0/2339452001":"2024-09-16T14:06:15.961693+0000","172.21.15.191:0/1984401962":"2024-09-16T14:05:10.469475+0000","172.21.15.191:0/420976848":"2024-09-16T14:07:13.859593+0000","172.21.15.191:0/1715384281":"2024-09-16T14:04:29.789163+0000","172.21.15.191:0/1452070078":"2024-09-16T14:04:21.768304+0000","172.21.15.191:0/922987045":"2024-09-16T14:08:02.996494+0000","172.21.15.191:0/3227952227":"2024-09-16T14:05:43.578504+0000","172.21.15.191:0/2307957520":"2024-09-16T14:05:01.841836+0000","172.21.15.191:0/1332979185":"2024-09-16T13:53:20.543598+0000","172.21.15.191:0/64250869":"2024-09-16T14:03:47.954472+0000","172.21.15.191:0/2542914986":"2024-09-16T14:05:34.742866+0000","172.21.15.191:0/3558354033":"2024-09-16T13:49:02.688221+0000","172.21.15.191:0/86987097":"2024-09-16T14:08:33.098286+0000","172.21.15.191:0/1645763072":"2024-09-16T13:49:58.410546+0000","172.21.15.191:0/388154838":"2024-09-16T13:48:06.986059+0000","172.21.15.191:0/502366332":"2024-09-16T14:04:53.018755+0000","172.21.15.191:0/3955414519":"2024-09-16T14:07:46.984976+0000","172.21.15.191:0/1079306742":"2024-09-16T14:06:32.858459+0000","172.21.15.191:0/3941490759":"2024-09-16T14:14:44.789936+0000","172.21.15.191:0/788785365":"2024-09-16T14:09:50.524412+0000","172.21.15.191:0/932669340":"2024-09-16T14:18:46.095210+0000","172.21.15.191:0/2608248056":"2024-09-16T14:04:12.900159+0000","172.21.15.191:0/3358173144":"2024-09-16T14:15:21.986894+0000","172.21.15.191:0/1436414":"2024-09-16T13:51:36.793453+0000","172.21.15.191:0/1983290075":"2024-09-16T13:46:24.776696+0000","172.21.15.191:0/3909433977":"2024-09-16T14:04:04.702316+0000","172.21.15.191:0/1857989703":"2024-09-16T14:07:22.605398+0000","172.21.15.191:0/1651357135":"2024-09-16T13:52:31.644838+0000","172.21.15.191:0/3587575345":"2024-09-16T14:03:56.683190+0000","172.21.15.191:0/632909243":"2024-09-16T14:06:24.859319+0000","172.21.15.191:0/1345841155":"2024-09-16T14:06:40.851469+0000","172.21.15.191:0/2428588659":"2024-09-16T14:06:07.762037+0000","172.21.15.191:0/2191780483":"2024-09-16T14:06:48.867661+0000","172.21.15.191:0/2404945880":"2024-09-16T14:06:57.073095+0000","172.21.15.191:0/3923198433":"2024-09-16T14:07:05.865372+0000","172.21.15.191:0/2418305813":"2024-09-16T14:09:36.544567+0000","172.21.15.191:0/1726136361":"2024-09-16T14:07:30.911035+0000","172.21.15.191:0/1589920012":"2024-09-16T14:07:38.988819+0000","172.21.15.191:0/3819829416":"2024-09-16T14:07:54.984663+0000","172.21.15.191:0/932746599":"2024-09-16T14:04:37.811197+0000","172.21.15.191:0/1915913838":"2024-09-16T14:08:54.442456+0000","172.21.15.191:0/1920517025":"2024-09-16T14:08:17.364976+0000","172.21.15.191:0/3522566784":"2024-09-16T14:08:24.453410+0000","172.21.15.191:0/3376647473":"2024-09-16T14:08:47.435363+0000","172.21.15.191:0/2025837665":"2024-09-16T14:10:06.628913+0000","172.21.15.191:0/1829408186":"2024-09-16T14:09:29.560050+0000","172.21.15.191:0/3864403648":"2024-09-16T14:09:08.464149+0000","172.21.15.191:0/3546350760":"2024-09-16T14:09:15.484255+0000","172.21.15.191:0/2457306634":"2024-09-16T14:09:22.494711+0000","172.21.15.191:0/2925010188":"2024-09-16T14:09:43.606212+0000","172.21.15.191:0/639914584":"2024-09-16T14:09:59.398239+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:30.235 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T13:19:30.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:19:30.492 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:30.493 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T13:19:30.504 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2024-09-16T13:19:29.956941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2798808,"kb_used_data":16884,"kb_used_omap":776,"kb_used_meta":2781111,"kb_avail":280316712,"statfs":{"total":289910292480,"available":287044313088,"internally_reserved":0,"allocated":17289216,"data_stored":9052327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":794673,"internal_metadata":2847858639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536168502,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-116387,"num_read_kb":-39395022,"num_write":-450762,"num_write_kb":-165027818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-12582912,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001419"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.703113+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T13:19:03.703113+0000","last_peered":"2024-09-16T13:19:03.703113+0000","last_clean":"2024-09-16T13:19:03.703113+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T13:19:03.703113+0000","last_undegraded":"2024-09-16T13:19:03.703113+0000","last_fullsized":"2024-09-16T13:19:03.703113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702979+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T13:19:03.702979+0000","last_peered":"2024-09-16T13:19:03.702979+0000","last_clean":"2024-09-16T13:19:03.702979+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T13:19:03.702979+0000","last_undegraded":"2024-09-16T13:19:03.702979+0000","last_fullsized":"2024-09-16T13:19:03.702979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702827+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T13:19:03.702827+0000","last_peered":"2024-09-16T13:19:03.702827+0000","last_clean":"2024-09-16T13:19:03.702827+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T13:19:03.702827+0000","last_undegraded":"2024-09-16T13:19:03.702827+0000","last_fullsized":"2024-09-16T13:19:03.702827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701814+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T13:19:03.701814+0000","last_peered":"2024-09-16T13:19:03.701814+0000","last_clean":"2024-09-16T13:19:03.701814+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T13:19:03.701814+0000","last_undegraded":"2024-09-16T13:19:03.701814+0000","last_fullsized":"2024-09-16T13:19:03.701814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.733130+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T13:19:24.733130+0000","last_peered":"2024-09-16T13:19:24.733130+0000","last_clean":"2024-09-16T13:19:24.733130+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T13:19:24.733130+0000","last_undegraded":"2024-09-16T13:19:24.733130+0000","last_fullsized":"2024-09-16T13:19:24.733130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.742282+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T13:19:24.742282+0000","last_peered":"2024-09-16T13:19:24.742282+0000","last_clean":"2024-09-16T13:19:24.742282+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T13:19:24.742282+0000","last_undegraded":"2024-09-16T13:19:24.742282+0000","last_fullsized":"2024-09-16T13:19:24.742282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.743311+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T13:19:24.743311+0000","last_peered":"2024-09-16T13:19:24.743311+0000","last_clean":"2024-09-16T13:19:24.743311+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T13:19:24.743311+0000","last_undegraded":"2024-09-16T13:19:24.743311+0000","last_fullsized":"2024-09-16T13:19:24.743311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":453,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701768+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T13:19:03.701768+0000","last_peered":"2024-09-16T13:19:03.701768+0000","last_clean":"2024-09-16T13:19:03.701768+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T13:19:03.701768+0000","last_undegraded":"2024-09-16T13:19:03.701768+0000","last_fullsized":"2024-09-16T13:19:03.701768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":517,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701255+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T13:19:03.701255+0000","last_peered":"2024-09-16T13:19:03.701255+0000","last_clean":"2024-09-16T13:19:03.701255+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T13:19:03.701255+0000","last_undegraded":"2024-09-16T13:19:03.701255+0000","last_fullsized":"2024-09-16T13:19:03.701255+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929164,"kb_used_data":5248,"kb_used_omap":250,"kb_used_meta":923653,"kb_avail":93442676,"statfs":{"total":96636764160,"available":95685300224,"internally_reserved":0,"allocated":5373952,"data_stored":2628332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256838,"internal_metadata":945820858},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":1.002,"15min":0.59399999999999997},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.14000000000000001},"max":{"1min":0.42999999999999999,"5min":50.787999999999997,"15min":50.787999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.96699999999999997,"15min":0.56299999999999994},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":0.47099999999999997,"5min":50.884999999999998,"15min":50.884999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.1429999999999998,"15min":12.419},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.45500000000000002,"5min":5.1100000000000003,"15min":8.8330000000000002},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935176,"kb_used_data":5820,"kb_used_omap":239,"kb_used_meta":929104,"kb_avail":93436664,"statfs":{"total":96636764160,"available":95679143936,"internally_reserved":0,"allocated":5959680,"data_stored":3212378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244816,"internal_metadata":951403440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.39000000000000001,"15min":0.34499999999999997},"min":{"1min":0.16300000000000001,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":5.5830000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.44700000000000001,"15min":0.36699999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":5.6589999999999998,"5min":5.6589999999999998,"15min":5.6589999999999998},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.41299999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":2.2709999999999999,"5min":4.7640000000000002,"15min":5.2859999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45500000000000002,"15min":0.40799999999999997},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.47199999999999998,"5min":7.9619999999999997,"15min":7.9619999999999997},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934468,"kb_used_data":5816,"kb_used_omap":286,"kb_used_meta":928353,"kb_avail":93437372,"statfs":{"total":96636764160,"available":95679868928,"internally_reserved":0,"allocated":5955584,"data_stored":3211617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":293019,"internal_metadata":950634341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.46500000000000002,"15min":0.379},"min":{"1min":0.22700000000000001,"5min":0.14599999999999999,"15min":0.13100000000000001},"max":{"1min":0.878,"5min":7.0970000000000004,"15min":8.6920000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.35799999999999998},"min":{"1min":0.22,"5min":0.19,"15min":0.186},"max":{"1min":0.89300000000000002,"5min":2.1720000000000002,"15min":3.4129999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.46999999999999997,"15min":0.40500000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14799999999999999},"max":{"1min":0.496,"5min":5.6509999999999998,"15min":13.260999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.438,"15min":0.36499999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.16},"max":{"1min":0.499,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:30.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:19:30.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:30.765 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T13:19:30.776 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2024-09-16T13:19:29.956941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2798808,"kb_used_data":16884,"kb_used_omap":776,"kb_used_meta":2781111,"kb_avail":280316712,"statfs":{"total":289910292480,"available":287044313088,"internally_reserved":0,"allocated":17289216,"data_stored":9052327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":794673,"internal_metadata":2847858639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536168502,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-116387,"num_read_kb":-39395022,"num_write":-450762,"num_write_kb":-165027818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-12582912,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001419"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.703113+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T13:19:03.703113+0000","last_peered":"2024-09-16T13:19:03.703113+0000","last_clean":"2024-09-16T13:19:03.703113+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T13:19:03.703113+0000","last_undegraded":"2024-09-16T13:19:03.703113+0000","last_fullsized":"2024-09-16T13:19:03.703113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702979+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T13:19:03.702979+0000","last_peered":"2024-09-16T13:19:03.702979+0000","last_clean":"2024-09-16T13:19:03.702979+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T13:19:03.702979+0000","last_undegraded":"2024-09-16T13:19:03.702979+0000","last_fullsized":"2024-09-16T13:19:03.702979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702827+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T13:19:03.702827+0000","last_peered":"2024-09-16T13:19:03.702827+0000","last_clean":"2024-09-16T13:19:03.702827+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T13:19:03.702827+0000","last_undegraded":"2024-09-16T13:19:03.702827+0000","last_fullsized":"2024-09-16T13:19:03.702827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701814+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T13:19:03.701814+0000","last_peered":"2024-09-16T13:19:03.701814+0000","last_clean":"2024-09-16T13:19:03.701814+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T13:19:03.701814+0000","last_undegraded":"2024-09-16T13:19:03.701814+0000","last_fullsized":"2024-09-16T13:19:03.701814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.733130+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T13:19:24.733130+0000","last_peered":"2024-09-16T13:19:24.733130+0000","last_clean":"2024-09-16T13:19:24.733130+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T13:19:24.733130+0000","last_undegraded":"2024-09-16T13:19:24.733130+0000","last_fullsized":"2024-09-16T13:19:24.733130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.742282+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T13:19:24.742282+0000","last_peered":"2024-09-16T13:19:24.742282+0000","last_clean":"2024-09-16T13:19:24.742282+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T13:19:24.742282+0000","last_undegraded":"2024-09-16T13:19:24.742282+0000","last_fullsized":"2024-09-16T13:19:24.742282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.743311+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T13:19:24.743311+0000","last_peered":"2024-09-16T13:19:24.743311+0000","last_clean":"2024-09-16T13:19:24.743311+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T13:19:24.743311+0000","last_undegraded":"2024-09-16T13:19:24.743311+0000","last_fullsized":"2024-09-16T13:19:24.743311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":453,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701768+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T13:19:03.701768+0000","last_peered":"2024-09-16T13:19:03.701768+0000","last_clean":"2024-09-16T13:19:03.701768+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T13:19:03.701768+0000","last_undegraded":"2024-09-16T13:19:03.701768+0000","last_fullsized":"2024-09-16T13:19:03.701768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":517,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701255+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T13:19:03.701255+0000","last_peered":"2024-09-16T13:19:03.701255+0000","last_clean":"2024-09-16T13:19:03.701255+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T13:19:03.701255+0000","last_undegraded":"2024-09-16T13:19:03.701255+0000","last_fullsized":"2024-09-16T13:19:03.701255+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929164,"kb_used_data":5248,"kb_used_omap":250,"kb_used_meta":923653,"kb_avail":93442676,"statfs":{"total":96636764160,"available":95685300224,"internally_reserved":0,"allocated":5373952,"data_stored":2628332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256838,"internal_metadata":945820858},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":1.002,"15min":0.59399999999999997},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.14000000000000001},"max":{"1min":0.42999999999999999,"5min":50.787999999999997,"15min":50.787999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.96699999999999997,"15min":0.56299999999999994},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":0.47099999999999997,"5min":50.884999999999998,"15min":50.884999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.1429999999999998,"15min":12.419},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.45500000000000002,"5min":5.1100000000000003,"15min":8.8330000000000002},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935176,"kb_used_data":5820,"kb_used_omap":239,"kb_used_meta":929104,"kb_avail":93436664,"statfs":{"total":96636764160,"available":95679143936,"internally_reserved":0,"allocated":5959680,"data_stored":3212378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244816,"internal_metadata":951403440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.39000000000000001,"15min":0.34499999999999997},"min":{"1min":0.16300000000000001,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":5.5830000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.44700000000000001,"15min":0.36699999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":5.6589999999999998,"5min":5.6589999999999998,"15min":5.6589999999999998},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.41299999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":2.2709999999999999,"5min":4.7640000000000002,"15min":5.2859999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45500000000000002,"15min":0.40799999999999997},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.47199999999999998,"5min":7.9619999999999997,"15min":7.9619999999999997},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934468,"kb_used_data":5816,"kb_used_omap":286,"kb_used_meta":928353,"kb_avail":93437372,"statfs":{"total":96636764160,"available":95679868928,"internally_reserved":0,"allocated":5955584,"data_stored":3211617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":293019,"internal_metadata":950634341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.46500000000000002,"15min":0.379},"min":{"1min":0.22700000000000001,"5min":0.14599999999999999,"15min":0.13100000000000001},"max":{"1min":0.878,"5min":7.0970000000000004,"15min":8.6920000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.35799999999999998},"min":{"1min":0.22,"5min":0.19,"15min":0.186},"max":{"1min":0.89300000000000002,"5min":2.1720000000000002,"15min":3.4129999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.46999999999999997,"15min":0.40500000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14799999999999999},"max":{"1min":0.496,"5min":5.6509999999999998,"15min":13.260999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.438,"15min":0.36499999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.16},"max":{"1min":0.499,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:30.777 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T13:19:30.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:19:31.033 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:31.033 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T13:19:31.045 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2024-09-16T13:19:29.956941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2798808,"kb_used_data":16884,"kb_used_omap":776,"kb_used_meta":2781111,"kb_avail":280316712,"statfs":{"total":289910292480,"available":287044313088,"internally_reserved":0,"allocated":17289216,"data_stored":9052327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":794673,"internal_metadata":2847858639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536168502,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-116387,"num_read_kb":-39395022,"num_write":-450762,"num_write_kb":-165027818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-12582912,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001419"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.703113+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T13:19:03.703113+0000","last_peered":"2024-09-16T13:19:03.703113+0000","last_clean":"2024-09-16T13:19:03.703113+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T13:19:03.703113+0000","last_undegraded":"2024-09-16T13:19:03.703113+0000","last_fullsized":"2024-09-16T13:19:03.703113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702979+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T13:19:03.702979+0000","last_peered":"2024-09-16T13:19:03.702979+0000","last_clean":"2024-09-16T13:19:03.702979+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T13:19:03.702979+0000","last_undegraded":"2024-09-16T13:19:03.702979+0000","last_fullsized":"2024-09-16T13:19:03.702979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702827+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T13:19:03.702827+0000","last_peered":"2024-09-16T13:19:03.702827+0000","last_clean":"2024-09-16T13:19:03.702827+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T13:19:03.702827+0000","last_undegraded":"2024-09-16T13:19:03.702827+0000","last_fullsized":"2024-09-16T13:19:03.702827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701814+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T13:19:03.701814+0000","last_peered":"2024-09-16T13:19:03.701814+0000","last_clean":"2024-09-16T13:19:03.701814+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T13:19:03.701814+0000","last_undegraded":"2024-09-16T13:19:03.701814+0000","last_fullsized":"2024-09-16T13:19:03.701814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.733130+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T13:19:24.733130+0000","last_peered":"2024-09-16T13:19:24.733130+0000","last_clean":"2024-09-16T13:19:24.733130+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T13:19:24.733130+0000","last_undegraded":"2024-09-16T13:19:24.733130+0000","last_fullsized":"2024-09-16T13:19:24.733130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.742282+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T13:19:24.742282+0000","last_peered":"2024-09-16T13:19:24.742282+0000","last_clean":"2024-09-16T13:19:24.742282+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T13:19:24.742282+0000","last_undegraded":"2024-09-16T13:19:24.742282+0000","last_fullsized":"2024-09-16T13:19:24.742282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.743311+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T13:19:24.743311+0000","last_peered":"2024-09-16T13:19:24.743311+0000","last_clean":"2024-09-16T13:19:24.743311+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T13:19:24.743311+0000","last_undegraded":"2024-09-16T13:19:24.743311+0000","last_fullsized":"2024-09-16T13:19:24.743311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":453,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701768+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T13:19:03.701768+0000","last_peered":"2024-09-16T13:19:03.701768+0000","last_clean":"2024-09-16T13:19:03.701768+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T13:19:03.701768+0000","last_undegraded":"2024-09-16T13:19:03.701768+0000","last_fullsized":"2024-09-16T13:19:03.701768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":517,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701255+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T13:19:03.701255+0000","last_peered":"2024-09-16T13:19:03.701255+0000","last_clean":"2024-09-16T13:19:03.701255+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T13:19:03.701255+0000","last_undegraded":"2024-09-16T13:19:03.701255+0000","last_fullsized":"2024-09-16T13:19:03.701255+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929164,"kb_used_data":5248,"kb_used_omap":250,"kb_used_meta":923653,"kb_avail":93442676,"statfs":{"total":96636764160,"available":95685300224,"internally_reserved":0,"allocated":5373952,"data_stored":2628332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256838,"internal_metadata":945820858},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":1.002,"15min":0.59399999999999997},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.14000000000000001},"max":{"1min":0.42999999999999999,"5min":50.787999999999997,"15min":50.787999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.96699999999999997,"15min":0.56299999999999994},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":0.47099999999999997,"5min":50.884999999999998,"15min":50.884999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.1429999999999998,"15min":12.419},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.45500000000000002,"5min":5.1100000000000003,"15min":8.8330000000000002},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935176,"kb_used_data":5820,"kb_used_omap":239,"kb_used_meta":929104,"kb_avail":93436664,"statfs":{"total":96636764160,"available":95679143936,"internally_reserved":0,"allocated":5959680,"data_stored":3212378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244816,"internal_metadata":951403440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.39000000000000001,"15min":0.34499999999999997},"min":{"1min":0.16300000000000001,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":5.5830000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.44700000000000001,"15min":0.36699999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":5.6589999999999998,"5min":5.6589999999999998,"15min":5.6589999999999998},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.41299999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":2.2709999999999999,"5min":4.7640000000000002,"15min":5.2859999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45500000000000002,"15min":0.40799999999999997},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.47199999999999998,"5min":7.9619999999999997,"15min":7.9619999999999997},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934468,"kb_used_data":5816,"kb_used_omap":286,"kb_used_meta":928353,"kb_avail":93437372,"statfs":{"total":96636764160,"available":95679868928,"internally_reserved":0,"allocated":5955584,"data_stored":3211617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":293019,"internal_metadata":950634341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.46500000000000002,"15min":0.379},"min":{"1min":0.22700000000000001,"5min":0.14599999999999999,"15min":0.13100000000000001},"max":{"1min":0.878,"5min":7.0970000000000004,"15min":8.6920000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.35799999999999998},"min":{"1min":0.22,"5min":0.19,"15min":0.186},"max":{"1min":0.89300000000000002,"5min":2.1720000000000002,"15min":3.4129999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.46999999999999997,"15min":0.40500000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14799999999999999},"max":{"1min":0.496,"5min":5.6509999999999998,"15min":13.260999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.438,"15min":0.36499999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.16},"max":{"1min":0.499,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:31.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T13:19:31.306 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:31.306 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":455,"fsid":"15053695-24d8-4e94-adfc-94f39586fb95","created":"2024-09-16T12:42:58.439707+0000","modified":"2024-09-16T13:19:28.731527+0000","last_up_change":"2024-09-16T12:43:07.180410+0000","last_in_change":"2024-09-16T12:42:59.237369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:43:07.438466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:43:13.318047+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"486ac1d9-0c28-4a08-9b32-aa19b63c78d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":33180},{"type":"v1","addr":"172.21.15.191:6801","nonce":33180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":33180},{"type":"v1","addr":"172.21.15.191:6803","nonce":33180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":33180},{"type":"v1","addr":"172.21.15.191:6807","nonce":33180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":33180},{"type":"v1","addr":"172.21.15.191:6805","nonce":33180}]},"public_addr":"172.21.15.191:6801/33180","cluster_addr":"172.21.15.191:6803/33180","heartbeat_back_addr":"172.21.15.191:6807/33180","heartbeat_front_addr":"172.21.15.191:6805/33180","state":["exists","up"]},{"osd":1,"uuid":"7526a6c2-0fe2-4a9b-bd1b-9d9718e65d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":33181},{"type":"v1","addr":"172.21.15.191:6812","nonce":33181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":33181},{"type":"v1","addr":"172.21.15.191:6816","nonce":33181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":33181},{"type":"v1","addr":"172.21.15.191:6823","nonce":33181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":33181},{"type":"v1","addr":"172.21.15.191:6820","nonce":33181}]},"public_addr":"172.21.15.191:6812/33181","cluster_addr":"172.21.15.191:6816/33181","heartbeat_back_addr":"172.21.15.191:6823/33181","heartbeat_front_addr":"172.21.15.191:6820/33181","state":["exists","up"]},{"osd":2,"uuid":"9894349c-b123-4be0-be82-1720ff2b1d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":33182},{"type":"v1","addr":"172.21.15.191:6809","nonce":33182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6811","nonce":33182},{"type":"v1","addr":"172.21.15.191:6813","nonce":33182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":33182},{"type":"v1","addr":"172.21.15.191:6821","nonce":33182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":33182},{"type":"v1","addr":"172.21.15.191:6817","nonce":33182}]},"public_addr":"172.21.15.191:6809/33182","cluster_addr":"172.21.15.191:6813/33182","heartbeat_back_addr":"172.21.15.191:6821/33182","heartbeat_front_addr":"172.21.15.191:6817/33182","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.365008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.385578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:43:04.340888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:0/4013810156":"2024-09-16T14:18:00.816387+0000","172.21.15.191:0/1193464417":"2024-09-16T14:17:23.902409+0000","172.21.15.191:0/3070994826":"2024-09-16T14:16:47.337609+0000","172.21.15.191:0/910748197":"2024-09-16T14:16:03.861334+0000","172.21.15.191:0/1238304107":"2024-09-16T14:14:07.778628+0000","172.21.15.191:0/1862466608":"2024-09-16T14:10:13.658094+0000","172.21.15.191:0/2594442942":"2024-09-16T14:09:01.613262+0000","172.21.15.191:0/2765424310":"2024-09-16T14:05:59.722885+0000","172.21.15.191:0/1455260247":"2024-09-16T14:19:24.264655+0000","172.21.15.191:0/1945055819":"2024-09-16T14:13:25.678957+0000","172.21.15.191:0/4266714643":"2024-09-16T13:45:39.236851+0000","172.21.15.191:0/3084174430":"2024-09-16T14:08:10.999696+0000","172.21.15.191:0/2189733109":"2024-09-16T14:05:51.704200+0000","172.21.15.191:0/1630258868":"2024-09-16T14:05:26.538888+0000","172.21.15.191:0/4255407941":"2024-09-16T13:50:47.225084+0000","172.21.15.191:0/2639548626":"2024-09-16T14:05:18.501394+0000","172.21.15.191:0/2705217819":"2024-09-16T14:04:44.153373+0000","172.21.15.191:0/3654451968":"2024-09-16T13:47:12.403445+0000","172.21.15.191:0/3433548665":"2024-09-16T14:08:41.036573+0000","172.21.15.191:0/2339452001":"2024-09-16T14:06:15.961693+0000","172.21.15.191:0/1984401962":"2024-09-16T14:05:10.469475+0000","172.21.15.191:0/420976848":"2024-09-16T14:07:13.859593+0000","172.21.15.191:0/1715384281":"2024-09-16T14:04:29.789163+0000","172.21.15.191:0/1452070078":"2024-09-16T14:04:21.768304+0000","172.21.15.191:0/922987045":"2024-09-16T14:08:02.996494+0000","172.21.15.191:0/3227952227":"2024-09-16T14:05:43.578504+0000","172.21.15.191:0/2307957520":"2024-09-16T14:05:01.841836+0000","172.21.15.191:0/1332979185":"2024-09-16T13:53:20.543598+0000","172.21.15.191:0/64250869":"2024-09-16T14:03:47.954472+0000","172.21.15.191:0/2542914986":"2024-09-16T14:05:34.742866+0000","172.21.15.191:0/3558354033":"2024-09-16T13:49:02.688221+0000","172.21.15.191:0/86987097":"2024-09-16T14:08:33.098286+0000","172.21.15.191:0/1645763072":"2024-09-16T13:49:58.410546+0000","172.21.15.191:0/388154838":"2024-09-16T13:48:06.986059+0000","172.21.15.191:0/502366332":"2024-09-16T14:04:53.018755+0000","172.21.15.191:0/3955414519":"2024-09-16T14:07:46.984976+0000","172.21.15.191:0/1079306742":"2024-09-16T14:06:32.858459+0000","172.21.15.191:0/3941490759":"2024-09-16T14:14:44.789936+0000","172.21.15.191:0/788785365":"2024-09-16T14:09:50.524412+0000","172.21.15.191:0/932669340":"2024-09-16T14:18:46.095210+0000","172.21.15.191:0/2608248056":"2024-09-16T14:04:12.900159+0000","172.21.15.191:0/3358173144":"2024-09-16T14:15:21.986894+0000","172.21.15.191:0/1436414":"2024-09-16T13:51:36.793453+0000","172.21.15.191:0/1983290075":"2024-09-16T13:46:24.776696+0000","172.21.15.191:0/3909433977":"2024-09-16T14:04:04.702316+0000","172.21.15.191:0/1857989703":"2024-09-16T14:07:22.605398+0000","172.21.15.191:0/1651357135":"2024-09-16T13:52:31.644838+0000","172.21.15.191:0/3587575345":"2024-09-16T14:03:56.683190+0000","172.21.15.191:0/632909243":"2024-09-16T14:06:24.859319+0000","172.21.15.191:0/1345841155":"2024-09-16T14:06:40.851469+0000","172.21.15.191:0/2428588659":"2024-09-16T14:06:07.762037+0000","172.21.15.191:0/2191780483":"2024-09-16T14:06:48.867661+0000","172.21.15.191:0/2404945880":"2024-09-16T14:06:57.073095+0000","172.21.15.191:0/3923198433":"2024-09-16T14:07:05.865372+0000","172.21.15.191:0/2418305813":"2024-09-16T14:09:36.544567+0000","172.21.15.191:0/1726136361":"2024-09-16T14:07:30.911035+0000","172.21.15.191:0/1589920012":"2024-09-16T14:07:38.988819+0000","172.21.15.191:0/3819829416":"2024-09-16T14:07:54.984663+0000","172.21.15.191:0/932746599":"2024-09-16T14:04:37.811197+0000","172.21.15.191:0/1915913838":"2024-09-16T14:08:54.442456+0000","172.21.15.191:0/1920517025":"2024-09-16T14:08:17.364976+0000","172.21.15.191:0/3522566784":"2024-09-16T14:08:24.453410+0000","172.21.15.191:0/3376647473":"2024-09-16T14:08:47.435363+0000","172.21.15.191:0/2025837665":"2024-09-16T14:10:06.628913+0000","172.21.15.191:0/1829408186":"2024-09-16T14:09:29.560050+0000","172.21.15.191:0/3864403648":"2024-09-16T14:09:08.464149+0000","172.21.15.191:0/3546350760":"2024-09-16T14:09:15.484255+0000","172.21.15.191:0/2457306634":"2024-09-16T14:09:22.494711+0000","172.21.15.191:0/2925010188":"2024-09-16T14:09:43.606212+0000","172.21.15.191:0/639914584":"2024-09-16T14:09:59.398239+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:32.319 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-16T13:19:32.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:32.449 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-16T13:19:32.449 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:32.449 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-16T13:19:32.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-16T13:19:32.718 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:19:32.730 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-16T13:19:32.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:32.846 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-16T13:19:32.846 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:32.846 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-16T13:19:32.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-16T13:19:33.119 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:19:33.131 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-16T13:19:33.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:33.248 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-16T13:19:33.249 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:33.249 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-16T13:19:33.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-16T13:19:33.517 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:19:33.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:19:33.787 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:33.787 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T13:19:33.801 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1545,"stamp":"2024-09-16T13:19:31.957291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2798808,"kb_used_data":16884,"kb_used_omap":776,"kb_used_meta":2781111,"kb_avail":280316712,"statfs":{"total":289910292480,"available":287044313088,"internally_reserved":0,"allocated":17289216,"data_stored":9052327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":794673,"internal_metadata":2847858639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536168502,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-116387,"num_read_kb":-39395022,"num_write":-450762,"num_write_kb":-165027818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-12582912,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.232095"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.703113+0000","last_change":"2024-09-16T12:43:17.216993+0000","last_active":"2024-09-16T13:19:03.703113+0000","last_peered":"2024-09-16T13:19:03.703113+0000","last_clean":"2024-09-16T13:19:03.703113+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T13:19:03.703113+0000","last_undegraded":"2024-09-16T13:19:03.703113+0000","last_fullsized":"2024-09-16T13:19:03.703113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:03.702581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702979+0000","last_change":"2024-09-16T12:43:17.217284+0000","last_active":"2024-09-16T13:19:03.702979+0000","last_peered":"2024-09-16T13:19:03.702979+0000","last_clean":"2024-09-16T13:19:03.702979+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T13:19:03.702979+0000","last_undegraded":"2024-09-16T13:19:03.702979+0000","last_fullsized":"2024-09-16T13:19:03.702979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:14.396659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.702827+0000","last_change":"2024-09-16T12:43:17.217128+0000","last_active":"2024-09-16T13:19:03.702827+0000","last_peered":"2024-09-16T13:19:03.702827+0000","last_clean":"2024-09-16T13:19:03.702827+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T13:19:03.702827+0000","last_undegraded":"2024-09-16T13:19:03.702827+0000","last_fullsized":"2024-09-16T13:19:03.702827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:45:00.200389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701814+0000","last_change":"2024-09-16T12:43:17.217380+0000","last_active":"2024-09-16T13:19:03.701814+0000","last_peered":"2024-09-16T13:19:03.701814+0000","last_clean":"2024-09-16T13:19:03.701814+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T13:19:03.701814+0000","last_undegraded":"2024-09-16T13:19:03.701814+0000","last_fullsized":"2024-09-16T13:19:03.701814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:03.596159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.733130+0000","last_change":"2024-09-16T12:43:17.214538+0000","last_active":"2024-09-16T13:19:24.733130+0000","last_peered":"2024-09-16T13:19:24.733130+0000","last_clean":"2024-09-16T13:19:24.733130+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T13:19:24.733130+0000","last_undegraded":"2024-09-16T13:19:24.733130+0000","last_fullsized":"2024-09-16T13:19:24.733130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:32.144488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.742282+0000","last_change":"2024-09-16T12:43:17.345558+0000","last_active":"2024-09-16T13:19:24.742282+0000","last_peered":"2024-09-16T13:19:24.742282+0000","last_clean":"2024-09-16T13:19:24.742282+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T13:19:24.742282+0000","last_undegraded":"2024-09-16T13:19:24.742282+0000","last_fullsized":"2024-09-16T13:19:24.742282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:06.137266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T13:19:24.743311+0000","last_change":"2024-09-16T12:43:17.345541+0000","last_active":"2024-09-16T13:19:24.743311+0000","last_peered":"2024-09-16T13:19:24.743311+0000","last_clean":"2024-09-16T13:19:24.743311+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T13:19:24.743311+0000","last_undegraded":"2024-09-16T13:19:24.743311+0000","last_fullsized":"2024-09-16T13:19:24.743311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:50.441307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":453,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701768+0000","last_change":"2024-09-16T12:43:17.216861+0000","last_active":"2024-09-16T13:19:03.701768+0000","last_peered":"2024-09-16T13:19:03.701768+0000","last_clean":"2024-09-16T13:19:03.701768+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T13:19:03.701768+0000","last_undegraded":"2024-09-16T13:19:03.701768+0000","last_fullsized":"2024-09-16T13:19:03.701768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:14.194803+0000","last_clean_scrub_stamp":"2024-09-16T12:43:14.194803+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:26.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":517,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.701255+0000","last_change":"2024-09-16T12:43:09.332865+0000","last_active":"2024-09-16T13:19:03.701255+0000","last_peered":"2024-09-16T13:19:03.701255+0000","last_clean":"2024-09-16T13:19:03.701255+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T13:19:03.701255+0000","last_undegraded":"2024-09-16T13:19:03.701255+0000","last_fullsized":"2024-09-16T13:19:03.701255+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:43:08.181099+0000","last_clean_scrub_stamp":"2024-09-16T12:43:08.181099+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:03.399414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929164,"kb_used_data":5248,"kb_used_omap":250,"kb_used_meta":923653,"kb_avail":93442676,"statfs":{"total":96636764160,"available":95685300224,"internally_reserved":0,"allocated":5373952,"data_stored":2628332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256838,"internal_metadata":945820858},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":1.002,"15min":0.59399999999999997},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.14000000000000001},"max":{"1min":0.42999999999999999,"5min":50.787999999999997,"15min":50.787999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.96699999999999997,"15min":0.56299999999999994},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":0.47099999999999997,"5min":50.884999999999998,"15min":50.884999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.1429999999999998,"15min":12.419},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.45500000000000002,"5min":5.1100000000000003,"15min":8.8330000000000002},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935176,"kb_used_data":5820,"kb_used_omap":239,"kb_used_meta":929104,"kb_avail":93436664,"statfs":{"total":96636764160,"available":95679143936,"internally_reserved":0,"allocated":5959680,"data_stored":3212378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244816,"internal_metadata":951403440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.39000000000000001,"15min":0.34499999999999997},"min":{"1min":0.16300000000000001,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":5.5830000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.44700000000000001,"15min":0.36699999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":5.6589999999999998,"5min":5.6589999999999998,"15min":5.6589999999999998},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.41299999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":2.2709999999999999,"5min":4.7640000000000002,"15min":5.2859999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45500000000000002,"15min":0.40799999999999997},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.47199999999999998,"5min":7.9619999999999997,"15min":7.9619999999999997},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934468,"kb_used_data":5816,"kb_used_omap":286,"kb_used_meta":928353,"kb_avail":93437372,"statfs":{"total":96636764160,"available":95679868928,"internally_reserved":0,"allocated":5955584,"data_stored":3211617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":293019,"internal_metadata":950634341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.46500000000000002,"15min":0.379},"min":{"1min":0.22700000000000001,"5min":0.14599999999999999,"15min":0.13100000000000001},"max":{"1min":0.878,"5min":7.0970000000000004,"15min":8.6920000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.35799999999999998},"min":{"1min":0.22,"5min":0.19,"15min":0.186},"max":{"1min":0.89300000000000002,"5min":2.1720000000000002,"15min":3.4129999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.46999999999999997,"15min":0.40500000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14799999999999999},"max":{"1min":0.496,"5min":5.6509999999999998,"15min":13.260999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.438,"15min":0.36499999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.16},"max":{"1min":0.499,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":0.23599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.805 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.806 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T12:43:14.194803+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.806 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T12:43:08.181099+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=43, tm_sec=8, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:33.806 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:19:53.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:19:54.080 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-16T13:19:54.081 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-16T13:19:54.092 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1556,"stamp":"2024-09-16T13:19:53.961139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2803024,"kb_used_data":16952,"kb_used_omap":776,"kb_used_meta":2785271,"kb_avail":280312496,"statfs":{"total":289910292480,"available":287039995904,"internally_reserved":0,"allocated":17358848,"data_stored":9086358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":794673,"internal_metadata":2852118479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001965"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.868724+0000","last_change":"2024-09-16T13:19:38.868699+0000","last_active":"2024-09-16T13:19:38.868724+0000","last_peered":"2024-09-16T13:19:38.868724+0000","last_clean":"2024-09-16T13:19:38.868724+0000","last_became_active":"2024-09-16T12:43:15.210618+0000","last_became_peered":"2024-09-16T12:43:15.210618+0000","last_unstale":"2024-09-16T13:19:38.868724+0000","last_undegraded":"2024-09-16T13:19:38.868724+0000","last_fullsized":"2024-09-16T13:19:38.868724+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:38.867751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:38.867751+0000","last_clean_scrub_stamp":"2024-09-16T13:19:38.867751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:47:56.982846+0000","scrub_duration":0.0014495370000000001,"objects_trimmed":0,"snaptrim_duration":0.00049443200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":464,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:44.935927+0000","last_change":"2024-09-16T13:19:44.935893+0000","last_active":"2024-09-16T13:19:44.935927+0000","last_peered":"2024-09-16T13:19:44.935927+0000","last_clean":"2024-09-16T13:19:44.935927+0000","last_became_active":"2024-09-16T12:43:15.210805+0000","last_became_peered":"2024-09-16T12:43:15.210805+0000","last_unstale":"2024-09-16T13:19:44.935927+0000","last_undegraded":"2024-09-16T13:19:44.935927+0000","last_fullsized":"2024-09-16T13:19:44.935927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:44.935741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:44.935741+0000","last_clean_scrub_stamp":"2024-09-16T13:19:44.935741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:38.239783+0000","scrub_duration":0.0023374369999999999,"objects_trimmed":0,"snaptrim_duration":0.00063436999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:41.872304+0000","last_change":"2024-09-16T13:19:41.872274+0000","last_active":"2024-09-16T13:19:41.872304+0000","last_peered":"2024-09-16T13:19:41.872304+0000","last_clean":"2024-09-16T13:19:41.872304+0000","last_became_active":"2024-09-16T12:43:15.210691+0000","last_became_peered":"2024-09-16T12:43:15.210691+0000","last_unstale":"2024-09-16T13:19:41.872304+0000","last_undegraded":"2024-09-16T13:19:41.872304+0000","last_fullsized":"2024-09-16T13:19:41.872304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:41.872159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:41.872159+0000","last_clean_scrub_stamp":"2024-09-16T13:19:41.872159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:21:15.515063+0000","scrub_duration":0.001547915,"objects_trimmed":0,"snaptrim_duration":0.00057575900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":464,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:42.902383+0000","last_change":"2024-09-16T13:19:42.902344+0000","last_active":"2024-09-16T13:19:42.902383+0000","last_peered":"2024-09-16T13:19:42.902383+0000","last_clean":"2024-09-16T13:19:42.902383+0000","last_became_active":"2024-09-16T12:43:15.209210+0000","last_became_peered":"2024-09-16T12:43:15.209210+0000","last_unstale":"2024-09-16T13:19:42.902383+0000","last_undegraded":"2024-09-16T13:19:42.902383+0000","last_fullsized":"2024-09-16T13:19:42.902383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:42.902234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:42.902234+0000","last_clean_scrub_stamp":"2024-09-16T13:19:42.902234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:57:36.537477+0000","scrub_duration":0.0014840249999999999,"objects_trimmed":0,"snaptrim_duration":0.00068619600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"455'3","reported_seq":469,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:33.762592+0000","last_change":"2024-09-16T13:19:33.762572+0000","last_active":"2024-09-16T13:19:33.762592+0000","last_peered":"2024-09-16T13:19:33.762592+0000","last_clean":"2024-09-16T13:19:33.762592+0000","last_became_active":"2024-09-16T12:43:15.209912+0000","last_became_peered":"2024-09-16T12:43:15.209912+0000","last_unstale":"2024-09-16T13:19:33.762592+0000","last_undegraded":"2024-09-16T13:19:33.762592+0000","last_fullsized":"2024-09-16T13:19:33.762592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'3","last_scrub_stamp":"2024-09-16T13:19:33.762501+0000","last_deep_scrub":"455'3","last_deep_scrub_stamp":"2024-09-16T13:19:33.762501+0000","last_clean_scrub_stamp":"2024-09-16T13:19:33.762501+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:55:04.851364+0000","scrub_duration":0.0027425129999999998,"objects_trimmed":0,"snaptrim_duration":0.00032832400000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:36.708914+0000","last_change":"2024-09-16T13:19:36.708884+0000","last_active":"2024-09-16T13:19:36.708914+0000","last_peered":"2024-09-16T13:19:36.708914+0000","last_clean":"2024-09-16T13:19:36.708914+0000","last_became_active":"2024-09-16T12:43:15.208768+0000","last_became_peered":"2024-09-16T12:43:15.208768+0000","last_unstale":"2024-09-16T13:19:36.708914+0000","last_undegraded":"2024-09-16T13:19:36.708914+0000","last_fullsized":"2024-09-16T13:19:36.708914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:36.708759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:36.708759+0000","last_clean_scrub_stamp":"2024-09-16T13:19:36.708759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:19.196162+0000","scrub_duration":0.0024539190000000002,"objects_trimmed":0,"snaptrim_duration":0.00027921400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:40.801625+0000","last_change":"2024-09-16T13:19:40.801594+0000","last_active":"2024-09-16T13:19:40.801625+0000","last_peered":"2024-09-16T13:19:40.801625+0000","last_clean":"2024-09-16T13:19:40.801625+0000","last_became_active":"2024-09-16T12:43:15.209671+0000","last_became_peered":"2024-09-16T12:43:15.209671+0000","last_unstale":"2024-09-16T13:19:40.801625+0000","last_undegraded":"2024-09-16T13:19:40.801625+0000","last_fullsized":"2024-09-16T13:19:40.801625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:40.801499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:40.801499+0000","last_clean_scrub_stamp":"2024-09-16T13:19:40.801499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:44:31.013114+0000","scrub_duration":0.0015386239999999999,"objects_trimmed":0,"snaptrim_duration":0.000288468,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"455'2","reported_seq":466,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:40.874190+0000","last_change":"2024-09-16T13:19:40.874149+0000","last_active":"2024-09-16T13:19:40.874190+0000","last_peered":"2024-09-16T13:19:40.874190+0000","last_clean":"2024-09-16T13:19:40.874190+0000","last_became_active":"2024-09-16T12:43:15.207771+0000","last_became_peered":"2024-09-16T12:43:15.207771+0000","last_unstale":"2024-09-16T13:19:40.874190+0000","last_undegraded":"2024-09-16T13:19:40.874190+0000","last_fullsized":"2024-09-16T13:19:40.874190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'2","last_scrub_stamp":"2024-09-16T13:19:40.874041+0000","last_deep_scrub":"455'2","last_deep_scrub_stamp":"2024-09-16T13:19:40.874041+0000","last_clean_scrub_stamp":"2024-09-16T13:19:40.874041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:36:02.364091+0000","scrub_duration":0.0024573189999999999,"objects_trimmed":0,"snaptrim_duration":0.00042440300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"455'54","reported_seq":531,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-16T13:19:35.838567+0000","last_change":"2024-09-16T13:19:35.838529+0000","last_active":"2024-09-16T13:19:35.838567+0000","last_peered":"2024-09-16T13:19:35.838567+0000","last_clean":"2024-09-16T13:19:35.838567+0000","last_became_active":"2024-09-16T12:43:09.331893+0000","last_became_peered":"2024-09-16T12:43:09.331893+0000","last_unstale":"2024-09-16T13:19:35.838567+0000","last_undegraded":"2024-09-16T13:19:35.838567+0000","last_fullsized":"2024-09-16T13:19:35.838567+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'54","last_scrub_stamp":"2024-09-16T13:19:35.838416+0000","last_deep_scrub":"455'54","last_deep_scrub_stamp":"2024-09-16T13:19:35.838416+0000","last_clean_scrub_stamp":"2024-09-16T13:19:35.838416+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:38.774676+0000","scrub_duration":0.0049894520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929172,"kb_used_data":5260,"kb_used_omap":250,"kb_used_meta":923653,"kb_avail":93442668,"statfs":{"total":96636764160,"available":95685292032,"internally_reserved":0,"allocated":5386240,"data_stored":2634941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256838,"internal_metadata":945820858},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":1.002,"15min":0.59399999999999997},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.14000000000000001},"max":{"1min":0.42999999999999999,"5min":50.787999999999997,"15min":50.787999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.96699999999999997,"15min":0.56299999999999994},"min":{"1min":0.20999999999999999,"5min":0.19600000000000001,"15min":0.16500000000000001},"max":{"1min":0.47099999999999997,"5min":50.884999999999998,"15min":50.884999999999998},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.1429999999999998,"15min":12.419},"last":0.374},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.45500000000000002,"5min":5.1100000000000003,"15min":8.8330000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935200,"kb_used_data":5848,"kb_used_omap":239,"kb_used_meta":929104,"kb_avail":93436640,"statfs":{"total":96636764160,"available":95679119360,"internally_reserved":0,"allocated":5988352,"data_stored":3226089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244816,"internal_metadata":951403440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.39000000000000001,"15min":0.34499999999999997},"min":{"1min":0.16300000000000001,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":5.5830000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.44700000000000001,"15min":0.36699999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":5.6589999999999998,"5min":5.6589999999999998,"15min":5.6589999999999998},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48699999999999999,"15min":0.41299999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":2.2709999999999999,"5min":4.7640000000000002,"15min":5.2859999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45500000000000002,"15min":0.40799999999999997},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.47199999999999998,"5min":7.9619999999999997,"15min":7.9619999999999997},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938652,"kb_used_data":5844,"kb_used_omap":286,"kb_used_meta":932513,"kb_avail":93433188,"statfs":{"total":96636764160,"available":95675584512,"internally_reserved":0,"allocated":5984256,"data_stored":3225328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":293019,"internal_metadata":954894181},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.46500000000000002,"15min":0.379},"min":{"1min":0.22700000000000001,"5min":0.14599999999999999,"15min":0.13100000000000001},"max":{"1min":0.878,"5min":7.0970000000000004,"15min":8.6920000000000002},"last":0.216},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.35799999999999998},"min":{"1min":0.22,"5min":0.19,"15min":0.186},"max":{"1min":0.89300000000000002,"5min":2.1720000000000002,"15min":3.4129999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.46999999999999997,"15min":0.40500000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14799999999999999},"max":{"1min":0.496,"5min":5.6509999999999998,"15min":13.260999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.438,"15min":0.36499999999999999},"min":{"1min":0.28000000000000003,"5min":0.19900000000000001,"15min":0.16},"max":{"1min":0.499,"5min":4.9080000000000004,"15min":4.9080000000000004},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:54.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-16T13:19:54.365 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:19:54.365 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:19:54.365 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-16T13:19:54.365 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:54.447 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-16T13:19:54.447 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-16T13:19:54.447 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:54.508 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-16T13:19:54.508 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-16T13:19:54.509 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:54.577 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-16T13:19:54.577 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:19:54.577 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-16T13:19:54.577 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:54.606 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-16T13:19:54.606 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:19:54.606 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-16T13:19:54.606 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:19:54.619 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-16T13:19:54.619 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:19:54.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:19:54.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T13:19:54.688 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-16T13:19:55.033 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T13:19:55.034 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-16T13:19:55.185 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T13:19:55.185 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-16T13:19:55.368 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:19:55.369 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/data/cluster2.mon.a.tgz 2024-09-16T13:19:55.370 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-16T13:19:55.385 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.MEbCV4hCcG 2024-09-16T13:19:55.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.MEbCV4hCcG 2024-09-16T13:19:55.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.MEbCV4hCcG 2024-09-16T13:19:55.644 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.MEbCV4hCcG is 477KB 2024-09-16T13:19:55.742 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.MEbCV4hCcG 2024-09-16T13:19:55.757 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:19:55.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:19:55.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:19:55.821 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:19:55.833 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-09-16T13:19:55.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T13:19:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:56.104 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":466,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T13:19:27.669999+0000","last_up_change":"2024-09-16T12:42:28.747107+0000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:42:29.075336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:42:34.363093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":32046},{"type":"v1","addr":"172.21.15.177:6809","nonce":32046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":32046},{"type":"v1","addr":"172.21.15.177:6811","nonce":32046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":32046},{"type":"v1","addr":"172.21.15.177:6815","nonce":32046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":32046},{"type":"v1","addr":"172.21.15.177:6813","nonce":32046}]},"public_addr":"172.21.15.177:6809/32046","cluster_addr":"172.21.15.177:6811/32046","heartbeat_back_addr":"172.21.15.177:6815/32046","heartbeat_front_addr":"172.21.15.177:6813/32046","state":["exists","up"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":32050},{"type":"v1","addr":"172.21.15.177:6817","nonce":32050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":32050},{"type":"v1","addr":"172.21.15.177:6819","nonce":32050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":32050},{"type":"v1","addr":"172.21.15.177:6823","nonce":32050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":32050},{"type":"v1","addr":"172.21.15.177:6821","nonce":32050}]},"public_addr":"172.21.15.177:6817/32050","cluster_addr":"172.21.15.177:6819/32050","heartbeat_back_addr":"172.21.15.177:6823/32050","heartbeat_front_addr":"172.21.15.177:6821/32050","state":["exists","up"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":32047},{"type":"v1","addr":"172.21.15.177:6801","nonce":32047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":32047},{"type":"v1","addr":"172.21.15.177:6803","nonce":32047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":32047},{"type":"v1","addr":"172.21.15.177:6807","nonce":32047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":32047},{"type":"v1","addr":"172.21.15.177:6805","nonce":32047}]},"public_addr":"172.21.15.177:6801/32047","cluster_addr":"172.21.15.177:6803/32047","heartbeat_back_addr":"172.21.15.177:6807/32047","heartbeat_front_addr":"172.21.15.177:6805/32047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.214807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.219527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.236130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:56.116 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T13:19:56.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:19:56.374 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:56.374 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T13:19:56.386 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-16T13:19:55.609467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929444,"kb_used_data":12412,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186076,"statfs":{"total":289910292480,"available":288958541824,"internally_reserved":0,"allocated":12709888,"data_stored":7343638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002376"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.508098+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T13:19:26.508098+0000","last_peered":"2024-09-16T13:19:26.508098+0000","last_clean":"2024-09-16T13:19:26.508098+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T13:19:26.508098+0000","last_undegraded":"2024-09-16T13:19:26.508098+0000","last_fullsized":"2024-09-16T13:19:26.508098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:11.040470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511938+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T13:19:26.511938+0000","last_peered":"2024-09-16T13:19:26.511938+0000","last_clean":"2024-09-16T13:19:26.511938+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T13:19:26.511938+0000","last_undegraded":"2024-09-16T13:19:26.511938+0000","last_fullsized":"2024-09-16T13:19:26.511938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:37.417191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.510736+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T13:19:26.510736+0000","last_peered":"2024-09-16T13:19:26.510736+0000","last_clean":"2024-09-16T13:19:26.510736+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T13:19:26.510736+0000","last_undegraded":"2024-09-16T13:19:26.510736+0000","last_fullsized":"2024-09-16T13:19:26.510736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:26.291723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504587+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T13:19:26.504587+0000","last_peered":"2024-09-16T13:19:26.504587+0000","last_clean":"2024-09-16T13:19:26.504587+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T13:19:26.504587+0000","last_undegraded":"2024-09-16T13:19:26.504587+0000","last_fullsized":"2024-09-16T13:19:26.504587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:45.836097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":467,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:25.677762+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T13:19:25.677762+0000","last_peered":"2024-09-16T13:19:25.677762+0000","last_clean":"2024-09-16T13:19:25.677762+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T13:19:25.677762+0000","last_undegraded":"2024-09-16T13:19:25.677762+0000","last_fullsized":"2024-09-16T13:19:25.677762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:34.320798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504580+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T13:19:26.504580+0000","last_peered":"2024-09-16T13:19:26.504580+0000","last_clean":"2024-09-16T13:19:26.504580+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T13:19:26.504580+0000","last_undegraded":"2024-09-16T13:19:26.504580+0000","last_fullsized":"2024-09-16T13:19:26.504580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.503833+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T13:19:26.503833+0000","last_peered":"2024-09-16T13:19:26.503833+0000","last_clean":"2024-09-16T13:19:26.503833+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T13:19:26.503833+0000","last_undegraded":"2024-09-16T13:19:26.503833+0000","last_fullsized":"2024-09-16T13:19:26.503833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511791+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T13:19:26.511791+0000","last_peered":"2024-09-16T13:19:26.511791+0000","last_clean":"2024-09-16T13:19:26.511791+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T13:19:26.511791+0000","last_undegraded":"2024-09-16T13:19:26.511791+0000","last_fullsized":"2024-09-16T13:19:26.511791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:51.045883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":530,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.502257+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T13:19:26.502257+0000","last_peered":"2024-09-16T13:19:26.502257+0000","last_clean":"2024-09-16T13:19:26.502257+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T13:19:26.502257+0000","last_undegraded":"2024-09-16T13:19:26.502257+0000","last_fullsized":"2024-09-16T13:19:26.502257+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.333013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313588,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058252,"statfs":{"total":96636764160,"available":96315650048,"internally_reserved":0,"allocated":3837952,"data_stored":2054288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30099999999999999,"15min":0.28799999999999998},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.11600000000000001},"max":{"1min":0.40400000000000003,"5min":0.434,"15min":1.1180000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.157},"max":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":1.1619999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.247,"15min":0.14099999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.91300000000000003},"last":0.309},{"interface":"front","average":{"1min":0.373,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.216,"15min":0.187},"max":{"1min":0.499,"5min":0.499,"15min":0.90100000000000002},"last":0.40300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.159},"max":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.192},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.49099999999999999},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.17599999999999999},"max":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.44500000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.35499999999999998},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.495},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314168,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94057672,"statfs":{"total":96636764160,"available":96315056128,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29399999999999998,"15min":0.28599999999999998},"min":{"1min":0.22500000000000001,"5min":0.183,"15min":0.14999999999999999},"max":{"1min":0.46400000000000002,"5min":0.47299999999999998,"15min":1.0189999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.318},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.46300000000000002,"5min":0.46400000000000002,"15min":1.4359999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.313},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.158},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.70799999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.187},"max":{"1min":0.48799999999999999,"5min":0.497,"15min":0.52700000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:56.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:19:56.647 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:56.647 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T13:19:56.659 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-16T13:19:55.609467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929444,"kb_used_data":12412,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186076,"statfs":{"total":289910292480,"available":288958541824,"internally_reserved":0,"allocated":12709888,"data_stored":7343638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002376"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.508098+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T13:19:26.508098+0000","last_peered":"2024-09-16T13:19:26.508098+0000","last_clean":"2024-09-16T13:19:26.508098+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T13:19:26.508098+0000","last_undegraded":"2024-09-16T13:19:26.508098+0000","last_fullsized":"2024-09-16T13:19:26.508098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:11.040470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511938+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T13:19:26.511938+0000","last_peered":"2024-09-16T13:19:26.511938+0000","last_clean":"2024-09-16T13:19:26.511938+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T13:19:26.511938+0000","last_undegraded":"2024-09-16T13:19:26.511938+0000","last_fullsized":"2024-09-16T13:19:26.511938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:37.417191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.510736+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T13:19:26.510736+0000","last_peered":"2024-09-16T13:19:26.510736+0000","last_clean":"2024-09-16T13:19:26.510736+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T13:19:26.510736+0000","last_undegraded":"2024-09-16T13:19:26.510736+0000","last_fullsized":"2024-09-16T13:19:26.510736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:26.291723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504587+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T13:19:26.504587+0000","last_peered":"2024-09-16T13:19:26.504587+0000","last_clean":"2024-09-16T13:19:26.504587+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T13:19:26.504587+0000","last_undegraded":"2024-09-16T13:19:26.504587+0000","last_fullsized":"2024-09-16T13:19:26.504587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:45.836097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":467,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:25.677762+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T13:19:25.677762+0000","last_peered":"2024-09-16T13:19:25.677762+0000","last_clean":"2024-09-16T13:19:25.677762+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T13:19:25.677762+0000","last_undegraded":"2024-09-16T13:19:25.677762+0000","last_fullsized":"2024-09-16T13:19:25.677762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:34.320798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504580+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T13:19:26.504580+0000","last_peered":"2024-09-16T13:19:26.504580+0000","last_clean":"2024-09-16T13:19:26.504580+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T13:19:26.504580+0000","last_undegraded":"2024-09-16T13:19:26.504580+0000","last_fullsized":"2024-09-16T13:19:26.504580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.503833+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T13:19:26.503833+0000","last_peered":"2024-09-16T13:19:26.503833+0000","last_clean":"2024-09-16T13:19:26.503833+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T13:19:26.503833+0000","last_undegraded":"2024-09-16T13:19:26.503833+0000","last_fullsized":"2024-09-16T13:19:26.503833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511791+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T13:19:26.511791+0000","last_peered":"2024-09-16T13:19:26.511791+0000","last_clean":"2024-09-16T13:19:26.511791+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T13:19:26.511791+0000","last_undegraded":"2024-09-16T13:19:26.511791+0000","last_fullsized":"2024-09-16T13:19:26.511791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:51.045883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":530,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.502257+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T13:19:26.502257+0000","last_peered":"2024-09-16T13:19:26.502257+0000","last_clean":"2024-09-16T13:19:26.502257+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T13:19:26.502257+0000","last_undegraded":"2024-09-16T13:19:26.502257+0000","last_fullsized":"2024-09-16T13:19:26.502257+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.333013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313588,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058252,"statfs":{"total":96636764160,"available":96315650048,"internally_reserved":0,"allocated":3837952,"data_stored":2054288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30099999999999999,"15min":0.28799999999999998},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.11600000000000001},"max":{"1min":0.40400000000000003,"5min":0.434,"15min":1.1180000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.157},"max":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":1.1619999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.247,"15min":0.14099999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.91300000000000003},"last":0.309},{"interface":"front","average":{"1min":0.373,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.216,"15min":0.187},"max":{"1min":0.499,"5min":0.499,"15min":0.90100000000000002},"last":0.40300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.159},"max":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.192},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.49099999999999999},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.17599999999999999},"max":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.44500000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.35499999999999998},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.495},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314168,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94057672,"statfs":{"total":96636764160,"available":96315056128,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29399999999999998,"15min":0.28599999999999998},"min":{"1min":0.22500000000000001,"5min":0.183,"15min":0.14999999999999999},"max":{"1min":0.46400000000000002,"5min":0.47299999999999998,"15min":1.0189999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.318},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.46300000000000002,"5min":0.46400000000000002,"15min":1.4359999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.313},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.158},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.70799999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.187},"max":{"1min":0.48799999999999999,"5min":0.497,"15min":0.52700000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:56.660 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T13:19:56.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:19:56.916 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:56.920 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T13:19:56.929 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-16T13:19:55.609467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929444,"kb_used_data":12412,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186076,"statfs":{"total":289910292480,"available":288958541824,"internally_reserved":0,"allocated":12709888,"data_stored":7343638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002376"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.508098+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T13:19:26.508098+0000","last_peered":"2024-09-16T13:19:26.508098+0000","last_clean":"2024-09-16T13:19:26.508098+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T13:19:26.508098+0000","last_undegraded":"2024-09-16T13:19:26.508098+0000","last_fullsized":"2024-09-16T13:19:26.508098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:11.040470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511938+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T13:19:26.511938+0000","last_peered":"2024-09-16T13:19:26.511938+0000","last_clean":"2024-09-16T13:19:26.511938+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T13:19:26.511938+0000","last_undegraded":"2024-09-16T13:19:26.511938+0000","last_fullsized":"2024-09-16T13:19:26.511938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:37.417191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.510736+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T13:19:26.510736+0000","last_peered":"2024-09-16T13:19:26.510736+0000","last_clean":"2024-09-16T13:19:26.510736+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T13:19:26.510736+0000","last_undegraded":"2024-09-16T13:19:26.510736+0000","last_fullsized":"2024-09-16T13:19:26.510736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:26.291723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504587+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T13:19:26.504587+0000","last_peered":"2024-09-16T13:19:26.504587+0000","last_clean":"2024-09-16T13:19:26.504587+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T13:19:26.504587+0000","last_undegraded":"2024-09-16T13:19:26.504587+0000","last_fullsized":"2024-09-16T13:19:26.504587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:45.836097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":467,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:25.677762+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T13:19:25.677762+0000","last_peered":"2024-09-16T13:19:25.677762+0000","last_clean":"2024-09-16T13:19:25.677762+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T13:19:25.677762+0000","last_undegraded":"2024-09-16T13:19:25.677762+0000","last_fullsized":"2024-09-16T13:19:25.677762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:34.320798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504580+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T13:19:26.504580+0000","last_peered":"2024-09-16T13:19:26.504580+0000","last_clean":"2024-09-16T13:19:26.504580+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T13:19:26.504580+0000","last_undegraded":"2024-09-16T13:19:26.504580+0000","last_fullsized":"2024-09-16T13:19:26.504580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.503833+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T13:19:26.503833+0000","last_peered":"2024-09-16T13:19:26.503833+0000","last_clean":"2024-09-16T13:19:26.503833+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T13:19:26.503833+0000","last_undegraded":"2024-09-16T13:19:26.503833+0000","last_fullsized":"2024-09-16T13:19:26.503833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.511791+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T13:19:26.511791+0000","last_peered":"2024-09-16T13:19:26.511791+0000","last_clean":"2024-09-16T13:19:26.511791+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T13:19:26.511791+0000","last_undegraded":"2024-09-16T13:19:26.511791+0000","last_fullsized":"2024-09-16T13:19:26.511791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:51.045883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":530,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.502257+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T13:19:26.502257+0000","last_peered":"2024-09-16T13:19:26.502257+0000","last_clean":"2024-09-16T13:19:26.502257+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T13:19:26.502257+0000","last_undegraded":"2024-09-16T13:19:26.502257+0000","last_fullsized":"2024-09-16T13:19:26.502257+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:22.333013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313588,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058252,"statfs":{"total":96636764160,"available":96315650048,"internally_reserved":0,"allocated":3837952,"data_stored":2054288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30099999999999999,"15min":0.28799999999999998},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.11600000000000001},"max":{"1min":0.40400000000000003,"5min":0.434,"15min":1.1180000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.157},"max":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":1.1619999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.247,"15min":0.14099999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.91300000000000003},"last":0.309},{"interface":"front","average":{"1min":0.373,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.216,"15min":0.187},"max":{"1min":0.499,"5min":0.499,"15min":0.90100000000000002},"last":0.40300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.159},"max":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.192},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.49099999999999999},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.17599999999999999},"max":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.44500000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.35499999999999998},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.495},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314168,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94057672,"statfs":{"total":96636764160,"available":96315056128,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29399999999999998,"15min":0.28599999999999998},"min":{"1min":0.22500000000000001,"5min":0.183,"15min":0.14999999999999999},"max":{"1min":0.46400000000000002,"5min":0.47299999999999998,"15min":1.0189999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.318},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.46300000000000002,"5min":0.46400000000000002,"15min":1.4359999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.313},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.158},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.70799999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.187},"max":{"1min":0.48799999999999999,"5min":0.497,"15min":0.52700000000000002},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:56.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T13:19:57.191 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:57.191 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":466,"fsid":"a00e16fb-57d0-4f62-8d54-577a0bafcdb7","created":"2024-09-16T12:42:19.211595+0000","modified":"2024-09-16T13:19:27.669999+0000","last_up_change":"2024-09-16T12:42:28.747107+0000","last_in_change":"2024-09-16T12:42:20.100535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:42:29.075336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:42:34.363093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"75458b1f-ab53-455c-b7b7-e4581435a54c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":32046},{"type":"v1","addr":"172.21.15.177:6809","nonce":32046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":32046},{"type":"v1","addr":"172.21.15.177:6811","nonce":32046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":32046},{"type":"v1","addr":"172.21.15.177:6815","nonce":32046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":32046},{"type":"v1","addr":"172.21.15.177:6813","nonce":32046}]},"public_addr":"172.21.15.177:6809/32046","cluster_addr":"172.21.15.177:6811/32046","heartbeat_back_addr":"172.21.15.177:6815/32046","heartbeat_front_addr":"172.21.15.177:6813/32046","state":["exists","up"]},{"osd":1,"uuid":"295d1942-3818-4b37-a350-593db1429c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":32050},{"type":"v1","addr":"172.21.15.177:6817","nonce":32050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":32050},{"type":"v1","addr":"172.21.15.177:6819","nonce":32050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":32050},{"type":"v1","addr":"172.21.15.177:6823","nonce":32050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":32050},{"type":"v1","addr":"172.21.15.177:6821","nonce":32050}]},"public_addr":"172.21.15.177:6817/32050","cluster_addr":"172.21.15.177:6819/32050","heartbeat_back_addr":"172.21.15.177:6823/32050","heartbeat_front_addr":"172.21.15.177:6821/32050","state":["exists","up"]},{"osd":2,"uuid":"11e62303-e15c-4bc3-8531-080922db8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":32047},{"type":"v1","addr":"172.21.15.177:6801","nonce":32047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":32047},{"type":"v1","addr":"172.21.15.177:6803","nonce":32047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":32047},{"type":"v1","addr":"172.21.15.177:6807","nonce":32047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":32047},{"type":"v1","addr":"172.21.15.177:6805","nonce":32047}]},"public_addr":"172.21.15.177:6801/32047","cluster_addr":"172.21.15.177:6803/32047","heartbeat_back_addr":"172.21.15.177:6807/32047","heartbeat_front_addr":"172.21.15.177:6805/32047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.214807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.219527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:42:25.236130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:58.205 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-16T13:19:58.205 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:58.334 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T13:19:58.334 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:58.334 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T13:19:58.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-16T13:19:58.607 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:19:58.619 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-16T13:19:58.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:58.734 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T13:19:58.735 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:58.735 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T13:19:58.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-16T13:19:59.003 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:19:59.016 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-16T13:19:59.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:59.132 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T13:19:59.132 INFO:teuthology.orchestra.run.smithi177.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:59.132 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T13:19:59.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-16T13:19:59.403 INFO:teuthology.orchestra.run.smithi177.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:19:59.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:19:59.671 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:19:59.672 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T13:19:59.683 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-16T13:19:59.610304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929444,"kb_used_data":12412,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186076,"statfs":{"total":289910292480,"available":288958541824,"internally_reserved":0,"allocated":12709888,"data_stored":7343638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002398"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":466,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.470737+0000","last_change":"2024-09-16T12:42:38.089492+0000","last_active":"2024-09-16T13:19:29.470737+0000","last_peered":"2024-09-16T13:19:29.470737+0000","last_clean":"2024-09-16T13:19:29.470737+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T13:19:29.470737+0000","last_undegraded":"2024-09-16T13:19:29.470737+0000","last_fullsized":"2024-09-16T13:19:29.470737+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.473624+0000","last_change":"2024-09-16T12:42:38.089807+0000","last_active":"2024-09-16T13:19:29.473624+0000","last_peered":"2024-09-16T13:19:29.473624+0000","last_clean":"2024-09-16T13:19:29.473624+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T13:19:29.473624+0000","last_undegraded":"2024-09-16T13:19:29.473624+0000","last_fullsized":"2024-09-16T13:19:29.473624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.473274+0000","last_change":"2024-09-16T12:42:38.089531+0000","last_active":"2024-09-16T13:19:29.473274+0000","last_peered":"2024-09-16T13:19:29.473274+0000","last_clean":"2024-09-16T13:19:29.473274+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T13:19:29.473274+0000","last_undegraded":"2024-09-16T13:19:29.473274+0000","last_fullsized":"2024-09-16T13:19:29.473274+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.468925+0000","last_change":"2024-09-16T12:42:38.089148+0000","last_active":"2024-09-16T13:19:29.468925+0000","last_peered":"2024-09-16T13:19:29.468925+0000","last_clean":"2024-09-16T13:19:29.468925+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T13:19:29.468925+0000","last_undegraded":"2024-09-16T13:19:29.468925+0000","last_fullsized":"2024-09-16T13:19:29.468925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":468,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:30.000489+0000","last_change":"2024-09-16T12:42:38.087391+0000","last_active":"2024-09-16T13:19:30.000489+0000","last_peered":"2024-09-16T13:19:30.000489+0000","last_clean":"2024-09-16T13:19:30.000489+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T13:19:30.000489+0000","last_undegraded":"2024-09-16T13:19:30.000489+0000","last_fullsized":"2024-09-16T13:19:30.000489+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.504580+0000","last_change":"2024-09-16T12:42:38.084045+0000","last_active":"2024-09-16T13:19:26.504580+0000","last_peered":"2024-09-16T13:19:26.504580+0000","last_clean":"2024-09-16T13:19:26.504580+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T13:19:26.504580+0000","last_undegraded":"2024-09-16T13:19:26.504580+0000","last_fullsized":"2024-09-16T13:19:26.504580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:34.833695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-16T13:19:26.503833+0000","last_change":"2024-09-16T12:42:38.084061+0000","last_active":"2024-09-16T13:19:26.503833+0000","last_peered":"2024-09-16T13:19:26.503833+0000","last_clean":"2024-09-16T13:19:26.503833+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T13:19:26.503833+0000","last_undegraded":"2024-09-16T13:19:26.503833+0000","last_fullsized":"2024-09-16T13:19:26.503833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:28.065769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":467,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.472225+0000","last_change":"2024-09-16T12:42:38.089129+0000","last_active":"2024-09-16T13:19:29.472225+0000","last_peered":"2024-09-16T13:19:29.472225+0000","last_clean":"2024-09-16T13:19:29.472225+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T13:19:29.472225+0000","last_undegraded":"2024-09-16T13:19:29.472225+0000","last_fullsized":"2024-09-16T13:19:29.472225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:35.066421+0000","last_clean_scrub_stamp":"2024-09-16T12:42:35.066421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":531,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:29.472343+0000","last_change":"2024-09-16T12:42:31.240431+0000","last_active":"2024-09-16T13:19:29.472343+0000","last_peered":"2024-09-16T13:19:29.472343+0000","last_clean":"2024-09-16T13:19:29.472343+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T13:19:29.472343+0000","last_undegraded":"2024-09-16T13:19:29.472343+0000","last_fullsized":"2024-09-16T13:19:29.472343+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:42:30.056585+0000","last_clean_scrub_stamp":"2024-09-16T12:42:30.056585+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313588,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058252,"statfs":{"total":96636764160,"available":96315650048,"internally_reserved":0,"allocated":3837952,"data_stored":2054288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30099999999999999,"15min":0.28799999999999998},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.11600000000000001},"max":{"1min":0.40400000000000003,"5min":0.434,"15min":1.1180000000000001},"last":0.221},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.157},"max":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":1.1619999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.247,"15min":0.14099999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.91300000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.216,"15min":0.187},"max":{"1min":0.499,"5min":0.499,"15min":0.90100000000000002},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.159},"max":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.192},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.49099999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.17599999999999999},"max":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.44500000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.35499999999999998},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.495},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738823,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314168,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94057672,"statfs":{"total":96636764160,"available":96315056128,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29399999999999998,"15min":0.28599999999999998},"min":{"1min":0.22500000000000001,"5min":0.183,"15min":0.14999999999999999},"max":{"1min":0.46400000000000002,"5min":0.47299999999999998,"15min":1.0189999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.318},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.46300000000000002,"5min":0.46400000000000002,"15min":1.4359999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.313},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.158},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.70799999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.187},"max":{"1min":0.48799999999999999,"5min":0.497,"15min":0.52700000000000002},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:19:59.684 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.684 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T12:42:35.066421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=35, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T12:42:30.056585+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=42, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:59.685 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:20:19.687 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:20:19.957 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T13:20:19.958 INFO:teuthology.orchestra.run.smithi177.stderr:dumped all 2024-09-16T13:20:19.969 INFO:teuthology.orchestra.run.smithi177.stdout:{"pg_ready":true,"pg_map":{"version":1599,"stamp":"2024-09-16T13:20:19.614016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929428,"kb_used_data":12412,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186092,"statfs":{"total":289910292480,"available":288958558208,"internally_reserved":0,"allocated":12709888,"data_stored":7343638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002269"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:08.291965+0000","last_change":"2024-09-16T13:20:08.291934+0000","last_active":"2024-09-16T13:20:08.291965+0000","last_peered":"2024-09-16T13:20:08.291965+0000","last_clean":"2024-09-16T13:20:08.291965+0000","last_became_active":"2024-09-16T12:42:36.086198+0000","last_became_peered":"2024-09-16T12:42:36.086198+0000","last_unstale":"2024-09-16T13:20:08.291965+0000","last_undegraded":"2024-09-16T13:20:08.291965+0000","last_fullsized":"2024-09-16T13:20:08.291965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:08.291821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:08.291821+0000","last_clean_scrub_stamp":"2024-09-16T13:20:08.291821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:18:39.952913+0000","scrub_duration":0.001411619,"objects_trimmed":0,"snaptrim_duration":0.00058036600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":476,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:06.289561+0000","last_change":"2024-09-16T13:20:06.289517+0000","last_active":"2024-09-16T13:20:06.289561+0000","last_peered":"2024-09-16T13:20:06.289561+0000","last_clean":"2024-09-16T13:20:06.289561+0000","last_became_active":"2024-09-16T12:42:36.085661+0000","last_became_peered":"2024-09-16T12:42:36.085661+0000","last_unstale":"2024-09-16T13:20:06.289561+0000","last_undegraded":"2024-09-16T13:20:06.289561+0000","last_fullsized":"2024-09-16T13:20:06.289561+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:06.289421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:06.289421+0000","last_clean_scrub_stamp":"2024-09-16T13:20:06.289421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:36.589522+0000","scrub_duration":0.0015067399999999999,"objects_trimmed":0,"snaptrim_duration":0.000651455,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.298872+0000","last_change":"2024-09-16T13:20:03.298838+0000","last_active":"2024-09-16T13:20:03.298872+0000","last_peered":"2024-09-16T13:20:03.298872+0000","last_clean":"2024-09-16T13:20:03.298872+0000","last_became_active":"2024-09-16T12:42:36.087194+0000","last_became_peered":"2024-09-16T12:42:36.087194+0000","last_unstale":"2024-09-16T13:20:03.298872+0000","last_undegraded":"2024-09-16T13:20:03.298872+0000","last_fullsized":"2024-09-16T13:20:03.298872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:03.298749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:03.298749+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.298749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:59.578565+0000","scrub_duration":0.001403704,"objects_trimmed":0,"snaptrim_duration":0.00086617200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:07.264492+0000","last_change":"2024-09-16T13:20:07.264461+0000","last_active":"2024-09-16T13:20:07.264492+0000","last_peered":"2024-09-16T13:20:07.264492+0000","last_clean":"2024-09-16T13:20:07.264492+0000","last_became_active":"2024-09-16T12:42:36.086344+0000","last_became_peered":"2024-09-16T12:42:36.086344+0000","last_unstale":"2024-09-16T13:20:07.264492+0000","last_undegraded":"2024-09-16T13:20:07.264492+0000","last_fullsized":"2024-09-16T13:20:07.264492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:07.264372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:07.264372+0000","last_clean_scrub_stamp":"2024-09-16T13:20:07.264372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:49.268120+0000","scrub_duration":0.0012613329999999999,"objects_trimmed":0,"snaptrim_duration":0.00047185499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"466'3","reported_seq":479,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:02.164840+0000","last_change":"2024-09-16T13:20:02.164806+0000","last_active":"2024-09-16T13:20:02.164840+0000","last_peered":"2024-09-16T13:20:02.164840+0000","last_clean":"2024-09-16T13:20:02.164840+0000","last_became_active":"2024-09-16T12:42:36.086063+0000","last_became_peered":"2024-09-16T12:42:36.086063+0000","last_unstale":"2024-09-16T13:20:02.164840+0000","last_undegraded":"2024-09-16T13:20:02.164840+0000","last_fullsized":"2024-09-16T13:20:02.164840+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'3","last_scrub_stamp":"2024-09-16T13:20:02.164711+0000","last_deep_scrub":"466'3","last_deep_scrub_stamp":"2024-09-16T13:20:02.164711+0000","last_clean_scrub_stamp":"2024-09-16T13:20:02.164711+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:28.745833+0000","scrub_duration":0.0026687239999999999,"objects_trimmed":0,"snaptrim_duration":0.0010212299999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":477,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:02.338363+0000","last_change":"2024-09-16T13:20:02.338334+0000","last_active":"2024-09-16T13:20:02.338363+0000","last_peered":"2024-09-16T13:20:02.338363+0000","last_clean":"2024-09-16T13:20:02.338363+0000","last_became_active":"2024-09-16T12:42:36.088502+0000","last_became_peered":"2024-09-16T12:42:36.088502+0000","last_unstale":"2024-09-16T13:20:02.338363+0000","last_undegraded":"2024-09-16T13:20:02.338363+0000","last_fullsized":"2024-09-16T13:20:02.338363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:02.338238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:02.338238+0000","last_clean_scrub_stamp":"2024-09-16T13:20:02.338238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:12.022248+0000","scrub_duration":0.001413515,"objects_trimmed":0,"snaptrim_duration":0.00028569499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.339259+0000","last_change":"2024-09-16T13:20:03.339218+0000","last_active":"2024-09-16T13:20:03.339259+0000","last_peered":"2024-09-16T13:20:03.339259+0000","last_clean":"2024-09-16T13:20:03.339259+0000","last_became_active":"2024-09-16T12:42:36.088573+0000","last_became_peered":"2024-09-16T12:42:36.088573+0000","last_unstale":"2024-09-16T13:20:03.339259+0000","last_undegraded":"2024-09-16T13:20:03.339259+0000","last_fullsized":"2024-09-16T13:20:03.339259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:20:03.339098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:20:03.339098+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.339098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:09.273703+0000","scrub_duration":0.0020352809999999999,"objects_trimmed":0,"snaptrim_duration":0.00033916600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"466'2","reported_seq":478,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:20:05.295404+0000","last_change":"2024-09-16T13:20:05.295368+0000","last_active":"2024-09-16T13:20:05.295404+0000","last_peered":"2024-09-16T13:20:05.295404+0000","last_clean":"2024-09-16T13:20:05.295404+0000","last_became_active":"2024-09-16T12:42:36.086227+0000","last_became_peered":"2024-09-16T12:42:36.086227+0000","last_unstale":"2024-09-16T13:20:05.295404+0000","last_undegraded":"2024-09-16T13:20:05.295404+0000","last_fullsized":"2024-09-16T13:20:05.295404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'2","last_scrub_stamp":"2024-09-16T13:20:05.295239+0000","last_deep_scrub":"466'2","last_deep_scrub_stamp":"2024-09-16T13:20:05.295239+0000","last_clean_scrub_stamp":"2024-09-16T13:20:05.295239+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:19.525196+0000","scrub_duration":0.002661559,"objects_trimmed":0,"snaptrim_duration":0.000462874,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"466'54","reported_seq":543,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-16T13:19:59.372965+0000","last_change":"2024-09-16T13:19:59.372945+0000","last_active":"2024-09-16T13:19:59.372965+0000","last_peered":"2024-09-16T13:19:59.372965+0000","last_clean":"2024-09-16T13:19:59.372965+0000","last_became_active":"2024-09-16T12:42:31.239427+0000","last_became_peered":"2024-09-16T12:42:31.239427+0000","last_unstale":"2024-09-16T13:19:59.372965+0000","last_undegraded":"2024-09-16T13:19:59.372965+0000","last_fullsized":"2024-09-16T13:19:59.372965+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'54","last_scrub_stamp":"2024-09-16T13:19:59.372870+0000","last_deep_scrub":"466'54","last_deep_scrub_stamp":"2024-09-16T13:19:59.372870+0000","last_clean_scrub_stamp":"2024-09-16T13:19:59.372870+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:13.189785+0000","scrub_duration":0.0040032510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313580,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058260,"statfs":{"total":96636764160,"available":96315658240,"internally_reserved":0,"allocated":3837952,"data_stored":2054288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30099999999999999,"15min":0.28799999999999998},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.11600000000000001},"max":{"1min":0.40400000000000003,"5min":0.434,"15min":1.1180000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.157},"max":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":1.1619999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.247,"15min":0.14099999999999999},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.91300000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.28299999999999997,"5min":0.216,"15min":0.187},"max":{"1min":0.499,"5min":0.499,"15min":0.90100000000000002},"last":0.496}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301684,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070156,"statfs":{"total":96636764160,"available":96327839744,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.159},"max":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.192},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.49099999999999999},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.26000000000000001,"5min":0.23400000000000001,"15min":0.17599999999999999},"max":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.44500000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.35499999999999998},"min":{"1min":0.32700000000000001,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.495},"last":0.34000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738827,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314164,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94057676,"statfs":{"total":96636764160,"available":96315060224,"internally_reserved":0,"allocated":4435968,"data_stored":2644675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.29399999999999998,"15min":0.28599999999999998},"min":{"1min":0.22500000000000001,"5min":0.183,"15min":0.14999999999999999},"max":{"1min":0.46400000000000002,"5min":0.47299999999999998,"15min":1.0189999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.318},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.46300000000000002,"5min":0.46400000000000002,"15min":1.4359999999999999},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.313},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.158},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.70799999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.187},"max":{"1min":0.48799999999999999,"5min":0.497,"15min":0.52700000000000002},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:20:19.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-16T13:20:20.244 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:20:20.245 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:20:20.245 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-16T13:20:20.245 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:20.289 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-16T13:20:20.289 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-16T13:20:20.289 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:20.335 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-16T13:20:20.335 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-16T13:20:20.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:20.371 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-16T13:20:20.372 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:20:20.372 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-16T13:20:20.372 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:20.437 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-16T13:20:20.437 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:20:20.437 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-16T13:20:20.437 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:20.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-09-16T13:20:20.489 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-16T13:20:20.490 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:20:20.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:20:20.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T13:20:20.568 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-16T13:20:20.763 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T13:20:20.763 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-16T13:20:20.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T13:20:20.850 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-16T13:20:20.926 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:20:20.926 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/data/cluster1.mon.a.tgz 2024-09-16T13:20:20.927 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-16T13:20:20.945 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.3qZ9S92XaK 2024-09-16T13:20:20.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.3qZ9S92XaK 2024-09-16T13:20:21.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.3qZ9S92XaK 2024-09-16T13:20:21.188 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.3qZ9S92XaK is 419KB 2024-09-16T13:20:21.286 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.3qZ9S92XaK 2024-09-16T13:20:21.301 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:20:21.302 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:20:21.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:20:21.373 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T13:20:21.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:20:21.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:20:21.432 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:20:21.443 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:20:21.448 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T13:20:21.450 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi177/crash 2024-09-16T13:20:21.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:20:21.500 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi191/crash 2024-09-16T13:20:21.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:20:21.527 INFO:tasks.ceph:Compressing logs... 2024-09-16T13:20:21.527 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:20:21.545 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:20:21.565 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31707.log 2024-09-16T13:20:21.565 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37003.log 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37045.log 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.31707.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31707.log.gz 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.37003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36898.log 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37003.log.gz 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32113.log 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37045.log.gz 2024-09-16T13:20:21.587 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36898.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36898.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.32113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39354.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32113.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39522.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36373.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39354.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39522.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36241.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36771.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36373.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36877.log 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36241.log.gz 2024-09-16T13:20:21.588 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36771.log.gz 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36665.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36877.log.gz 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36322.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31946.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36665.log.gz 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36322.log: gzip 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.36322.log.gz -5 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.39396.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.31946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39565.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31946.log.gz 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39396.log.gz 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36538.log 2024-09-16T13:20:21.589 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36811.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39565.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36538.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36537.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39605.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36811.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36537.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36537.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32045.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39228.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39605.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.32045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32045.log.gz 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36751.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39312.log 2024-09-16T13:20:21.590 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39228.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37087.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36751.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36937.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39312.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.37087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37087.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36415.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36937.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36415.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.admin.36415.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.36297.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36297.log.gz 2024-09-16T13:20:21.591 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36539.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39438.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-mgr.x.log: /var/log/ceph/cluster1-client.admin.36539.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36539.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.39186.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39438.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31865.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39186.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39186.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.39270.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.31865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31865.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36157.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39270.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39270.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.39102.log 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36157.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36157.log.gz 2024-09-16T13:20:21.592 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/cluster1.audit.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39102.log.gz 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31905.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5/var/log/ceph/cluster1.audit.log: --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.31905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31905.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36625.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36664.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.36625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36625.log.gz 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27210.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36664.log.gz 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39060.log 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.tmp-client.admin.27210.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27210.log.gz 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr: 92.5% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-16T13:20:21.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39060.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39480.log 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39144.log 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39480.log: /var/log/ceph/cluster1.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39480.log.gz 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36457.log 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.39144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39144.log.gz 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cluster1-client.admin.36457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36457.log.gz 2024-09-16T13:20:21.594 INFO:teuthology.orchestra.run.smithi177.stderr: 86.9% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-16T13:20:21.598 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50550.log 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67286.log 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47852.log 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50451.log 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50550.log.gz 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57972.log 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67286.log.gz 2024-09-16T13:20:21.600 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40848.log 2024-09-16T13:20:21.601 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61329.log 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster2-client.mirror.40848.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72478.log 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50451.log.gz 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38588.log 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48853.log 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72478.log.gz 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38588.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57972.log.gz 2024-09-16T13:20:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63900.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38588.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69738.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48853.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61329.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8%/var/log/ceph/cluster2-client.mirror.63900.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48853.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62001.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61308.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69738.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72282.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62001.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63900.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43419.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61308.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72282.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62001.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73926.log 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72282.log.gz 2024-09-16T13:20:21.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47892.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43419.log: /var/log/ceph/cluster2-client.mirror.73926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68280.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73926.log.gz 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43419.log.gz 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37078.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47892.log.gz 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40848.log.gz/var/log/ceph/cluster2-client.mirror.68280.log: 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45314.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75819.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37078.log.gz 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66066.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75819.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75819.log.gz 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58040.log 2024-09-16T13:20:21.608 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.63123.log 2024-09-16T13:20:21.624 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66066.log: /var/log/ceph/cluster2-client.mirror.58040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55082.log 2024-09-16T13:20:21.624 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68280.log.gz 2024-09-16T13:20:21.624 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.63123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59232.log 2024-09-16T13:20:21.624 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58040.log.gz 2024-09-16T13:20:21.624 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54524.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55082.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose/var/log/ceph/cluster1-client.mirror.59232.log: -- /var/log/ceph/cluster2-client.mirror.49390.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54524.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50025.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59232.log.gz 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66066.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54524.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72027.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50708.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60870.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72027.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50708.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50025.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72604.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50708.log.gz 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster2-client.mirror.60870.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63297.log 2024-09-16T13:20:21.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72604.log: gzip -5 --verbose -- 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45314.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster2-client.mirror.66046.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63297.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72604.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65488.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63297.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.66046.log: --verbose -- /var/log/ceph/cluster2-client.mirror.65839.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60870.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65488.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66046.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53376.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65488.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52020.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65839.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.53376.log: --verbose -- /var/log/ceph/cluster2-client.mirror.50167.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54029.log 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53376.log.gz 2024-09-16T13:20:21.626 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52020.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67908.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50167.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42535.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74301.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61188.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54029.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75076.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61188.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74301.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40784.log 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61188.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75076.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67908.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75076.log.gz 2024-09-16T13:20:21.627 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43508.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61537.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40784.log.gz/var/log/ceph/cluster1-client.mirror.43508.log: 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70593.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40315.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.70593.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61537.log.gz 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 86.7% --verbose -- /var/log/ceph/cluster2-client.mirror.40805.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70593.log.gz 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62500.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40315.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45001.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40805.log: 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65009.log 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45001.log: /var/log/ceph/cluster1-client.mirror.62500.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62500.log.gz 2024-09-16T13:20:21.628 INFO:teuthology.orchestra.run.smithi191.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45001.log.gz 2024-09-16T13:20:21.629 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75162.log 2024-09-16T13:20:21.629 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.65009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46214.log 2024-09-16T13:20:21.629 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65009.log.gz 2024-09-16T13:20:21.629 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75162.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75162.log.gz 2024-09-16T13:20:21.632 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75118.log 2024-09-16T13:20:21.632 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43508.log.gz 2024-09-16T13:20:21.632 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63850.log 2024-09-16T13:20:21.632 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46214.log: /var/log/ceph/cluster1-client.mirror.75118.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75118.log.gz 2024-09-16T13:20:21.633 INFO:teuthology.orchestra.run.smithi191.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40805.log.gz 2024-09-16T13:20:21.664 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.63123.log.gz 2024-09-16T13:20:21.664 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46214.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69940.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64879.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63494.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63850.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69940.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69850.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72731.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64879.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69850.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63494.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57072.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69850.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49568.log 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72731.log.gz 2024-09-16T13:20:21.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59051.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49568.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57072.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48269.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49568.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51373.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59051.log: /var/log/ceph/cluster2-client.mirror.48269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40245.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59051.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48269.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52902.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53352.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40245.log: /var/log/ceph/cluster1-client.mirror.52902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58084.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42311.log 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52902.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53352.log.gz 2024-09-16T13:20:21.666 INFO:teuthology.orchestra.run.smithi191.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40245.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58084.log: gzip 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58084.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68995.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42311.log: gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42311.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster2-client.mirror.70759.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69648.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68995.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65148.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74257.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69648.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70759.log.gz/var/log/ceph/cluster2-client.mirror.65148.log: 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61762.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74257.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65148.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62753.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74257.log.gz 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72395.log 2024-09-16T13:20:21.667 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61762.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39131.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62753.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50184.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72395.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47048.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.39131.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.50184.log: --verbose -- /var/log/ceph/cluster1-client.mirror.41733.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64773.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.41733.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47048.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42491.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50184.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41733.log.gz 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67554.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48766.log 2024-09-16T13:20:21.668 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67554.log: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73299.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64773.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48766.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48766.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69063.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68725.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73299.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69063.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67554.log.gz 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69063.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68793.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75331.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68725.log: /var/log/ceph/cluster1-client.mirror.68793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68239.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68725.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75331.log: -- replaced with /var/log/ceph/cluster1-client.mirror.68793.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75331.log.gz 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66562.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58156.log 2024-09-16T13:20:21.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66562.log: 88.1%gzip 93.5% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.42491.log.gz --verbose 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster2-client.mirror.52000.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68239.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46022.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58156.log: /var/log/ceph/cluster2-client.mirror.52000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44108.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46022.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58156.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65612.log 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52000.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.44108.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66562.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.71985.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58407.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71985.log: /var/log/ceph/cluster2-client.mirror.65612.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65612.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71985.log.gz 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68032.log 2024-09-16T13:20:21.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55845.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68032.log: /var/log/ceph/cluster1-client.mirror.58407.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58407.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63691.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55845.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55845.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46022.log.gzgzip 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr: -5 90.5% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.68032.log.gz /var/log/ceph/cluster1-client.mirror.58383.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61465.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63691.log: /var/log/ceph/cluster1-client.mirror.58383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53220.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58383.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61465.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63691.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33247.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61465.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57703.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66604.log 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57703.log: /var/log/ceph/cluster2-client.admin.33247.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33247.log.gz 2024-09-16T13:20:21.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43551.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53220.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44524.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43551.log: 92.6% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66604.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57703.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76072.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74280.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44524.log: /var/log/ceph/cluster2-client.admin.76072.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76072.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43551.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52042.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53107.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52042.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74280.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44524.log.gz 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72537.log 2024-09-16T13:20:21.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49988.log 2024-09-16T13:20:21.673 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53107.log.gz 2024-09-16T13:20:21.673 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37543.log 2024-09-16T13:20:21.676 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53537.log 2024-09-16T13:20:21.677 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37543.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37543.log.gz 2024-09-16T13:20:21.683 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43656.log 2024-09-16T13:20:21.684 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53537.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53537.log.gz 2024-09-16T13:20:21.696 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37165.log 2024-09-16T13:20:21.697 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.43656.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43656.log.gz 2024-09-16T13:20:21.714 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66728.log 2024-09-16T13:20:21.714 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37165.log.gz 2024-09-16T13:20:21.722 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38187.log 2024-09-16T13:20:21.736 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66728.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66728.log.gz 2024-09-16T13:20:21.738 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60849.log 2024-09-16T13:20:21.739 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38187.log: 73.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38187.log.gz 2024-09-16T13:20:21.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40042.log 2024-09-16T13:20:21.751 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60849.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60849.log.gz 2024-09-16T13:20:21.760 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45635.log 2024-09-16T13:20:21.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40042.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40042.log.gz 2024-09-16T13:20:21.780 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73684.log 2024-09-16T13:20:21.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45635.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45635.log.gz 2024-09-16T13:20:21.799 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42359.log 2024-09-16T13:20:21.800 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73684.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73684.log.gz 2024-09-16T13:20:21.815 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76112.log 2024-09-16T13:20:21.816 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42359.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42359.log.gz 2024-09-16T13:20:21.825 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67120.log 2024-09-16T13:20:21.834 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.76112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42287.log 2024-09-16T13:20:21.837 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76112.log.gz 2024-09-16T13:20:21.842 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53059.log 2024-09-16T13:20:21.849 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42287.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67120.log.gz 2024-09-16T13:20:21.851 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56066.log 2024-09-16T13:20:21.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53059.log: 87.4% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42287.log.gz 2024-09-16T13:20:21.862 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60270.log 2024-09-16T13:20:21.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56066.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47852.log.gz 2024-09-16T13:20:21.864 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56066.log.gz 2024-09-16T13:20:21.864 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53059.log.gz 2024-09-16T13:20:21.864 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70345.log 2024-09-16T13:20:21.866 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60270.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60270.log.gz 2024-09-16T13:20:21.866 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74904.log 2024-09-16T13:20:21.866 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70345.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70345.log.gz 2024-09-16T13:20:21.866 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.49221.log 2024-09-16T13:20:21.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74904.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74904.log.gz 2024-09-16T13:20:21.867 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45764.log 2024-09-16T13:20:21.877 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.49221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72224.log 2024-09-16T13:20:21.883 INFO:teuthology.orchestra.run.smithi177.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-16T13:20:21.884 INFO:teuthology.orchestra.run.smithi177.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-16T13:20:21.891 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45764.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45764.log.gz 2024-09-16T13:20:21.894 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60162.log 2024-09-16T13:20:21.894 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72224.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72224.log.gz 2024-09-16T13:20:21.908 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48559.log 2024-09-16T13:20:21.909 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60162.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60162.log.gz 2024-09-16T13:20:21.923 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49469.log 2024-09-16T13:20:21.956 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49706.log 2024-09-16T13:20:21.965 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70779.log 2024-09-16T13:20:21.966 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49706.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72537.log.gz 2024-09-16T13:20:21.967 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68680.log 2024-09-16T13:20:21.968 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70779.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70779.log.gz 2024-09-16T13:20:21.968 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72201.log 2024-09-16T13:20:21.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68680.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68680.log.gz 2024-09-16T13:20:21.969 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.43720.log 2024-09-16T13:20:21.971 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72201.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72201.log.gz 2024-09-16T13:20:21.971 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42896.log 2024-09-16T13:20:21.973 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.43720.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.43720.log.gz 2024-09-16T13:20:21.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63629.log 2024-09-16T13:20:21.975 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42896.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42896.log.gz 2024-09-16T13:20:21.975 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42872.log 2024-09-16T13:20:21.976 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63629.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63629.log.gz 2024-09-16T13:20:21.977 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36913.log 2024-09-16T13:20:21.978 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42872.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42872.log.gz 2024-09-16T13:20:21.978 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37879.log 2024-09-16T13:20:21.980 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36913.log.gz 2024-09-16T13:20:21.980 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49887.log 2024-09-16T13:20:21.980 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37879.log.gz 2024-09-16T13:20:21.980 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42148.log 2024-09-16T13:20:21.981 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49887.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49887.log.gz 2024-09-16T13:20:21.982 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48328.log 2024-09-16T13:20:21.983 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42148.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42148.log.gz 2024-09-16T13:20:21.984 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55169.log 2024-09-16T13:20:21.985 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48328.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48328.log.gz 2024-09-16T13:20:21.985 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61130.log 2024-09-16T13:20:21.987 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55169.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55169.log.gz 2024-09-16T13:20:21.987 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60318.log 2024-09-16T13:20:21.989 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61130.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61130.log.gz 2024-09-16T13:20:21.989 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40568.log 2024-09-16T13:20:21.990 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60318.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60318.log.gz 2024-09-16T13:20:21.990 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59505.log 2024-09-16T13:20:21.991 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40568.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40568.log.gz 2024-09-16T13:20:21.991 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61371.log 2024-09-16T13:20:21.992 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59505.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59505.log.gz 2024-09-16T13:20:21.993 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72768.log 2024-09-16T13:20:21.994 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61371.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61371.log.gz 2024-09-16T13:20:21.994 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.54029.log 2024-09-16T13:20:21.995 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72768.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72768.log.gz 2024-09-16T13:20:21.996 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65102.log 2024-09-16T13:20:22.005 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.54029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61019.log 2024-09-16T13:20:22.010 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65102.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65102.log.gz 2024-09-16T13:20:22.016 INFO:teuthology.orchestra.run.smithi191.stderr: 94.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43467.log 2024-09-16T13:20:22.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61019.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61019.log.gz 2024-09-16T13:20:22.017 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.54029.log.gz 2024-09-16T13:20:22.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53777.log 2024-09-16T13:20:22.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43467.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43467.log.gz 2024-09-16T13:20:22.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60673.log 2024-09-16T13:20:22.028 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.72827.log 2024-09-16T13:20:22.032 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60673.log: 92.3% 93.8% 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60673.log.gz 2024-09-16T13:20:22.032 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53777.log.gz 2024-09-16T13:20:22.032 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.49221.log.gz 2024-09-16T13:20:22.033 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60801.log 2024-09-16T13:20:22.037 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.72827.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.72827.log.gz 2024-09-16T13:20:22.038 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48311.log 2024-09-16T13:20:22.039 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60801.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60801.log.gz 2024-09-16T13:20:22.039 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55886.log 2024-09-16T13:20:22.040 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48311.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48311.log.gz 2024-09-16T13:20:22.040 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41591.log 2024-09-16T13:20:22.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55886.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55886.log.gz 2024-09-16T13:20:22.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.53777.log 2024-09-16T13:20:22.045 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41591.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41591.log.gz 2024-09-16T13:20:22.045 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46557.log 2024-09-16T13:20:22.054 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.53777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39216.log 2024-09-16T13:20:22.060 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45655.log 2024-09-16T13:20:22.061 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39216.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46557.log.gz 2024-09-16T13:20:22.067 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.53777.log.gz 2024-09-16T13:20:22.068 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49075.log 2024-09-16T13:20:22.068 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45655.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39216.log.gz 2024-09-16T13:20:22.073 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45655.log.gz 2024-09-16T13:20:22.074 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45612.log 2024-09-16T13:20:22.075 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49075.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49075.log.gz 2024-09-16T13:20:22.075 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45721.log 2024-09-16T13:20:22.079 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45612.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45612.log.gz 2024-09-16T13:20:22.080 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36955.log 2024-09-16T13:20:22.085 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45721.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45721.log.gz 2024-09-16T13:20:22.086 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38612.log 2024-09-16T13:20:22.086 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36955.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36955.log.gz 2024-09-16T13:20:22.087 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60390.log 2024-09-16T13:20:22.087 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38612.log.gz 2024-09-16T13:20:22.088 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57369.log 2024-09-16T13:20:22.088 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60390.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60390.log.gz 2024-09-16T13:20:22.089 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75735.log 2024-09-16T13:20:22.089 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57369.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57369.log.gz 2024-09-16T13:20:22.090 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54139.log 2024-09-16T13:20:22.090 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75735.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75735.log.gz 2024-09-16T13:20:22.090 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62392.log 2024-09-16T13:20:22.091 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54139.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49390.log.gz 2024-09-16T13:20:22.092 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69963.log 2024-09-16T13:20:22.093 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62392.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54139.log.gz 2024-09-16T13:20:22.093 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75401.log 2024-09-16T13:20:22.094 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69963.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62392.log.gz 2024-09-16T13:20:22.094 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69963.log.gz 2024-09-16T13:20:22.094 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37711.log 2024-09-16T13:20:22.095 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75401.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75401.log.gz 2024-09-16T13:20:22.095 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74719.log 2024-09-16T13:20:22.095 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37711.log.gz 2024-09-16T13:20:22.096 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59691.log 2024-09-16T13:20:22.096 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74719.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74719.log.gz 2024-09-16T13:20:22.097 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54235.log 2024-09-16T13:20:22.098 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59691.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59691.log.gz 2024-09-16T13:20:22.098 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51799.log 2024-09-16T13:20:22.099 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54235.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54235.log.gz 2024-09-16T13:20:22.100 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62372.log 2024-09-16T13:20:22.109 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.51799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60608.log 2024-09-16T13:20:22.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62372.log: 92.5% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52042.log.gz 2024-09-16T13:20:22.122 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51406.log 2024-09-16T13:20:22.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60608.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60608.log.gz 2024-09-16T13:20:22.124 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62372.log.gz 2024-09-16T13:20:22.138 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49988.log.gz 2024-09-16T13:20:22.139 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38806.log 2024-09-16T13:20:22.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51406.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51406.log.gz 2024-09-16T13:20:22.141 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56609.log 2024-09-16T13:20:22.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.38806.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38806.log.gz 2024-09-16T13:20:22.142 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73550.log 2024-09-16T13:20:22.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56609.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56609.log.gz 2024-09-16T13:20:22.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52206.log 2024-09-16T13:20:22.146 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73550.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73550.log.gz 2024-09-16T13:20:22.146 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56867.log 2024-09-16T13:20:22.146 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52206.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52206.log.gz 2024-09-16T13:20:22.148 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53284.log 2024-09-16T13:20:22.148 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56867.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56867.log.gz 2024-09-16T13:20:22.148 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47028.log 2024-09-16T13:20:22.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53284.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53284.log.gz 2024-09-16T13:20:22.149 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42778.log 2024-09-16T13:20:22.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47028.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47028.log.gz 2024-09-16T13:20:22.151 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50629.log 2024-09-16T13:20:22.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42778.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42778.log.gz 2024-09-16T13:20:22.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51373.log.gz 2024-09-16T13:20:22.154 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64121.log 2024-09-16T13:20:22.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74846.log 2024-09-16T13:20:22.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50629.log: /var/log/ceph/cluster2-client.mirror.64121.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50629.log.gz 2024-09-16T13:20:22.155 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64121.log.gz 2024-09-16T13:20:22.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51604.log 2024-09-16T13:20:22.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62842.log 2024-09-16T13:20:22.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51604.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74846.log.gz 2024-09-16T13:20:22.158 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65984.log 2024-09-16T13:20:22.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62842.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51799.log.gz 2024-09-16T13:20:22.160 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51740.log 2024-09-16T13:20:22.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65984.log: 92.3% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65984.log.gz 2024-09-16T13:20:22.161 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62842.log.gz 2024-09-16T13:20:22.161 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59654.log 2024-09-16T13:20:22.162 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48949.log 2024-09-16T13:20:22.162 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51740.log.gz 2024-09-16T13:20:22.162 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59654.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42535.log.gz 2024-09-16T13:20:22.162 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57255.log 2024-09-16T13:20:22.163 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71964.log 2024-09-16T13:20:22.163 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59654.log.gz 2024-09-16T13:20:22.164 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38177.log 2024-09-16T13:20:22.164 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71964.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57255.log.gz 2024-09-16T13:20:22.164 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71964.log.gz 2024-09-16T13:20:22.164 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51037.log 2024-09-16T13:20:22.165 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53916.log 2024-09-16T13:20:22.165 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38177.log: /var/log/ceph/cluster2-client.mirror.51037.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51037.log.gz 2024-09-16T13:20:22.165 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44108.log.gz 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65756.log 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36781.log 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53916.log.gz 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41565.log 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36781.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36781.log.gz 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45999.log 2024-09-16T13:20:22.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.41565.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41565.log.gz 2024-09-16T13:20:22.180 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55633.log 2024-09-16T13:20:22.180 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45999.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65756.log.gz 2024-09-16T13:20:22.180 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56516.log 2024-09-16T13:20:22.180 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55633.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38177.log.gz 2024-09-16T13:20:22.181 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55633.log.gz 2024-09-16T13:20:22.182 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57811.log 2024-09-16T13:20:22.182 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43052.log 2024-09-16T13:20:22.183 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56516.log: /var/log/ceph/cluster2-client.mirror.57811.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56516.log.gz 2024-09-16T13:20:22.183 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57811.log.gz 2024-09-16T13:20:22.183 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45850.log 2024-09-16T13:20:22.184 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43052.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45999.log.gz 2024-09-16T13:20:22.185 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41685.log 2024-09-16T13:20:22.185 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41410.log 2024-09-16T13:20:22.185 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41685.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41685.log.gz 2024-09-16T13:20:22.188 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43052.log.gz 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47793.log 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45357.log/var/log/ceph/cluster2-client.mirror.41410.log: 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45850.log.gz 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47793.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47793.log.gz 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68905.log 2024-09-16T13:20:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52585.log 2024-09-16T13:20:22.193 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68905.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68905.log.gz 2024-09-16T13:20:22.193 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72690.log 2024-09-16T13:20:22.194 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52585.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52585.log.gz 2024-09-16T13:20:22.194 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44881.log 2024-09-16T13:20:22.195 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72690.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72690.log.gz 2024-09-16T13:20:22.195 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68260.log 2024-09-16T13:20:22.195 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41410.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44881.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44881.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45357.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70008.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57301.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70008.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68260.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51624.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70008.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48537.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57301.log: /var/log/ceph/cluster2-client.mirror.51624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74407.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51624.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48537.log: gzip -5 --verbose 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57301.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster2-client.mirror.63043.log 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74407.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48537.log.gz 2024-09-16T13:20:22.199 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68322.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73985.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74407.log.gz 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68322.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63043.log.gz 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57835.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68322.log.gz 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66996.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46517.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.66996.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57835.log.gz 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62926.log 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46517.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46517.log.gz 2024-09-16T13:20:22.206 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55749.log 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62926.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62926.log.gz 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66996.log.gz 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54572.log 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46845.log 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55749.log.gz 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64920.log 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54572.log.gz 2024-09-16T13:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46845.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46845.log.gz 2024-09-16T13:20:22.208 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46151.log 2024-09-16T13:20:22.208 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57052.log 2024-09-16T13:20:22.212 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46151.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46151.log.gz 2024-09-16T13:20:22.213 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62705.log 2024-09-16T13:20:22.214 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57052.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57052.log.gz 2024-09-16T13:20:22.214 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53698.log 2024-09-16T13:20:22.215 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62705.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62705.log.gz 2024-09-16T13:20:22.215 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69153.log 2024-09-16T13:20:22.217 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53698.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53698.log.gz 2024-09-16T13:20:22.217 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74696.log 2024-09-16T13:20:22.218 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69153.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69153.log.gz 2024-09-16T13:20:22.218 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47385.log 2024-09-16T13:20:22.219 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74696.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74696.log.gz 2024-09-16T13:20:22.219 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70390.log 2024-09-16T13:20:22.220 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.47385.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47385.log.gz 2024-09-16T13:20:22.220 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71855.log 2024-09-16T13:20:22.221 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70390.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70390.log.gz 2024-09-16T13:20:22.221 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61561.log 2024-09-16T13:20:22.222 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71855.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71855.log.gz 2024-09-16T13:20:22.222 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33040.log 2024-09-16T13:20:22.223 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61561.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61561.log.gz 2024-09-16T13:20:22.223 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65057.log 2024-09-16T13:20:22.224 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.33040.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33040.log.gz 2024-09-16T13:20:22.224 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44609.log 2024-09-16T13:20:22.225 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.65057.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65057.log.gz 2024-09-16T13:20:22.225 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57010.log 2024-09-16T13:20:22.226 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44609.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44609.log.gz 2024-09-16T13:20:22.227 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69018.log 2024-09-16T13:20:22.228 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57010.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57010.log.gz 2024-09-16T13:20:22.228 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67513.log 2024-09-16T13:20:22.229 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69018.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69018.log.gz 2024-09-16T13:20:22.229 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64016.log 2024-09-16T13:20:22.230 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67513.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67513.log.gz 2024-09-16T13:20:22.231 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68860.log 2024-09-16T13:20:22.231 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64016.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64016.log.gz 2024-09-16T13:20:22.232 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59256.log 2024-09-16T13:20:22.232 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68860.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68860.log.gz 2024-09-16T13:20:22.232 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56384.log 2024-09-16T13:20:22.233 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59256.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59256.log.gz 2024-09-16T13:20:22.233 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73148.log 2024-09-16T13:20:22.234 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56384.log: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64920.log.gz 2024-09-16T13:20:22.235 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56384.log.gz 2024-09-16T13:20:22.236 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53557.log 2024-09-16T13:20:22.236 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67534.log 2024-09-16T13:20:22.237 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73148.log: /var/log/ceph/cluster2-client.mirror.53557.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53557.log.gz 2024-09-16T13:20:22.244 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52657.log 2024-09-16T13:20:22.246 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67534.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67534.log.gz 2024-09-16T13:20:22.246 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38636.log 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52657.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52657.log.gz 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46417.log 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38636.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.38636.log.gz 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60912.log 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46417.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46417.log.gz 2024-09-16T13:20:22.249 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52830.log 2024-09-16T13:20:22.251 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60912.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60912.log.gz 2024-09-16T13:20:22.251 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68883.log 2024-09-16T13:20:22.253 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52830.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52830.log.gz 2024-09-16T13:20:22.253 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67472.log 2024-09-16T13:20:22.253 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68883.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68883.log.gz 2024-09-16T13:20:22.253 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39114.log 2024-09-16T13:20:22.255 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67472.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67472.log.gz 2024-09-16T13:20:22.255 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54396.log 2024-09-16T13:20:22.255 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.39114.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.39114.log.gz 2024-09-16T13:20:22.256 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60825.log 2024-09-16T13:20:22.258 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54396.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54396.log.gz 2024-09-16T13:20:22.258 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45893.log 2024-09-16T13:20:22.259 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60825.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60825.log.gz 2024-09-16T13:20:22.260 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61624.log 2024-09-16T13:20:22.260 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45893.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48949.log.gz 2024-09-16T13:20:22.264 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45893.log.gz 2024-09-16T13:20:22.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69693.log 2024-09-16T13:20:22.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44567.log 2024-09-16T13:20:22.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69693.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69693.log.gz 2024-09-16T13:20:22.267 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61624.log.gz 2024-09-16T13:20:22.267 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50730.log 2024-09-16T13:20:22.267 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38721.log 2024-09-16T13:20:22.272 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44567.log: /var/log/ceph/cluster2-client.mirror.50730.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44567.log.gz 2024-09-16T13:20:22.273 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52878.log 2024-09-16T13:20:22.274 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38721.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38721.log.gz 2024-09-16T13:20:22.274 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65880.log 2024-09-16T13:20:22.275 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52878.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52878.log.gz 2024-09-16T13:20:22.275 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44431.log 2024-09-16T13:20:22.278 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65880.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65880.log.gz 2024-09-16T13:20:22.278 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69400.log 2024-09-16T13:20:22.279 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44431.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44431.log.gz 2024-09-16T13:20:22.279 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43395.log 2024-09-16T13:20:22.280 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69400.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69400.log.gz 2024-09-16T13:20:22.281 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68928.log 2024-09-16T13:20:22.282 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43395.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43395.log.gz 2024-09-16T13:20:22.282 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73089.log 2024-09-16T13:20:22.283 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68928.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68928.log.gz 2024-09-16T13:20:22.283 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50045.log 2024-09-16T13:20:22.284 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73089.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73089.log.gz 2024-09-16T13:20:22.285 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69895.log 2024-09-16T13:20:22.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50045.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50045.log.gz 2024-09-16T13:20:22.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54765.log 2024-09-16T13:20:22.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69895.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69895.log.gz 2024-09-16T13:20:22.287 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49627.log 2024-09-16T13:20:22.288 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54765.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54765.log.gz 2024-09-16T13:20:22.288 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57634.log 2024-09-16T13:20:22.300 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49627.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49706.log.gz 2024-09-16T13:20:22.301 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58108.log 2024-09-16T13:20:22.301 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57634.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57634.log.gz 2024-09-16T13:20:22.303 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62159.log 2024-09-16T13:20:22.303 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58108.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58108.log.gz 2024-09-16T13:20:22.303 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68198.log 2024-09-16T13:20:22.306 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62159.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62159.log.gz 2024-09-16T13:20:22.306 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49331.log 2024-09-16T13:20:22.307 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68198.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68198.log.gz 2024-09-16T13:20:22.307 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51443.log 2024-09-16T13:20:22.307 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49331.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49331.log.gz 2024-09-16T13:20:22.308 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67244.log 2024-09-16T13:20:22.309 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51443.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51443.log.gz 2024-09-16T13:20:22.309 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74071.log 2024-09-16T13:20:22.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67244.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67244.log.gz 2024-09-16T13:20:22.312 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60995.log 2024-09-16T13:20:22.313 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74071.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74071.log.gz 2024-09-16T13:20:22.314 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.41130.log 2024-09-16T13:20:22.315 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60995.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60995.log.gz 2024-09-16T13:20:22.315 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69198.log 2024-09-16T13:20:22.326 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.41130.log: 92.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74784.log 2024-09-16T13:20:22.327 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69198.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69198.log.gz 2024-09-16T13:20:22.327 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.41130.log.gz 2024-09-16T13:20:22.327 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48153.log 2024-09-16T13:20:22.329 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74784.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74784.log.gz 2024-09-16T13:20:22.329 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37437.log 2024-09-16T13:20:22.329 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48153.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48153.log.gz 2024-09-16T13:20:22.331 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53487.log 2024-09-16T13:20:22.331 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37437.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37437.log.gz 2024-09-16T13:20:22.331 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52681.log 2024-09-16T13:20:22.332 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53487.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53487.log.gz 2024-09-16T13:20:22.332 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63763.log 2024-09-16T13:20:22.334 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52681.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52681.log.gz 2024-09-16T13:20:22.334 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44260.log 2024-09-16T13:20:22.334 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63763.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63763.log.gz 2024-09-16T13:20:22.335 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44363.log 2024-09-16T13:20:22.336 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.44260.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44260.log.gz 2024-09-16T13:20:22.336 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58742.log 2024-09-16T13:20:22.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44363.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51604.log.gz 2024-09-16T13:20:22.337 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44363.log.gz 2024-09-16T13:20:22.339 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73067.log 2024-09-16T13:20:22.339 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.58742.log: -- /var/log/ceph/cluster1-client.mirror.56131.log 2024-09-16T13:20:22.340 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58742.log.gz 2024-09-16T13:20:22.340 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61786.log 2024-09-16T13:20:22.341 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56131.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73067.log.gz 2024-09-16T13:20:22.341 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56131.log.gz 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70615.log 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57907.log 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61786.log.gz/var/log/ceph/cluster1-client.mirror.70615.log: 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70615.log.gz 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60951.log 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63920.log 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57907.log.gz 2024-09-16T13:20:22.346 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.50284.log 2024-09-16T13:20:22.347 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63920.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63920.log.gz 2024-09-16T13:20:22.347 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60951.log.gz 2024-09-16T13:20:22.347 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52947.log 2024-09-16T13:20:22.347 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.50284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45420.log 2024-09-16T13:20:22.347 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52947.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52947.log.gz 2024-09-16T13:20:22.348 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41887.log 2024-09-16T13:20:22.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45420.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45420.log.gz 2024-09-16T13:20:22.354 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50844.log 2024-09-16T13:20:22.355 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41887.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41887.log.gz 2024-09-16T13:20:22.355 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56319.log 2024-09-16T13:20:22.356 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50844.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50844.log.gz 2024-09-16T13:20:22.356 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43009.log 2024-09-16T13:20:22.357 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56319.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56319.log.gz 2024-09-16T13:20:22.357 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65259.log 2024-09-16T13:20:22.362 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43009.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43009.log.gz 2024-09-16T13:20:22.363 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55496.log 2024-09-16T13:20:22.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65259.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65259.log.gz 2024-09-16T13:20:22.366 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62303.log 2024-09-16T13:20:22.366 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55496.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55496.log.gz 2024-09-16T13:20:22.367 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73169.log 2024-09-16T13:20:22.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62303.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62303.log.gz 2024-09-16T13:20:22.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65736.log 2024-09-16T13:20:22.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73169.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73169.log.gz 2024-09-16T13:20:22.369 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63715.log 2024-09-16T13:20:22.369 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65736.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65736.log.gz 2024-09-16T13:20:22.370 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45956.log 2024-09-16T13:20:22.370 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63715.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63715.log.gz 2024-09-16T13:20:22.371 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45979.log 2024-09-16T13:20:22.376 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45956.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45956.log.gz 2024-09-16T13:20:22.377 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56629.log 2024-09-16T13:20:22.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45979.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45979.log.gz 2024-09-16T13:20:22.383 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38271.log 2024-09-16T13:20:22.385 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56629.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56629.log.gz 2024-09-16T13:20:22.385 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47226.log 2024-09-16T13:20:22.395 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.38271.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38271.log.gz 2024-09-16T13:20:22.395 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64261.log 2024-09-16T13:20:22.396 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47226.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47226.log.gz 2024-09-16T13:20:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32867.log 2024-09-16T13:20:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64261.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64261.log.gz 2024-09-16T13:20:22.398 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55207.log 2024-09-16T13:20:22.398 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32867.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32867.log.gz 2024-09-16T13:20:22.398 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72827.log 2024-09-16T13:20:22.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55207.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55207.log.gz 2024-09-16T13:20:22.400 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38772.log 2024-09-16T13:20:22.400 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72827.log: 82.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72827.log.gz 2024-09-16T13:20:22.402 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72495.log 2024-09-16T13:20:22.402 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38772.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38772.log.gz 2024-09-16T13:20:22.402 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49469.log.gz 2024-09-16T13:20:22.402 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.42148.log 2024-09-16T13:20:22.402 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51905.log 2024-09-16T13:20:22.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.42148.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73985.log.gz 2024-09-16T13:20:22.404 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72495.log.gz 2024-09-16T13:20:22.404 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74387.log 2024-09-16T13:20:22.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.72282.log 2024-09-16T13:20:22.406 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74387.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51905.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74387.log.gz 2024-09-16T13:20:22.406 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.406 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55388.log 2024-09-16T13:20:22.406 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.2.72282.log: -- /var/log/ceph/cluster2-client.mirror.62456.log 2024-09-16T13:20:22.408 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55388.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55388.log.gz 2024-09-16T13:20:22.408 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69535.log 2024-09-16T13:20:22.410 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62456.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62456.log.gz 2024-09-16T13:20:22.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61738.log 2024-09-16T13:20:22.410 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69535.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.72282.log.gz 2024-09-16T13:20:22.410 INFO:teuthology.orchestra.run.smithi191.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69535.log.gz 2024-09-16T13:20:22.410 INFO:teuthology.orchestra.run.smithi191.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.50284.log.gz 2024-09-16T13:20:22.414 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68455.log 2024-09-16T13:20:22.414 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72515.log 2024-09-16T13:20:22.414 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68455.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61738.log.gz 2024-09-16T13:20:22.414 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60453.log 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68455.log.gz 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75777.log 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72515.log: /var/log/ceph/cluster2-client.mirror.60453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38091.log 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75777.log: 0.0% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72515.log.gz 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.75777.log.gz 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60453.log.gz 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59325.log 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.38091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67575.log 2024-09-16T13:20:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73774.log 2024-09-16T13:20:22.426 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67575.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67575.log.gz 2024-09-16T13:20:22.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68478.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59325.log.gz 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73774.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38091.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59164.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68478.log: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42335.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73774.log.gz 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68478.log.gz 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45225.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59164.log: /var/log/ceph/cluster1-client.mirror.42335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73257.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45225.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42335.log.gz 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45225.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57562.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59164.log.gz 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56203.log 2024-09-16T13:20:22.427 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73257.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49526.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57562.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48074.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49410.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49526.log: /var/log/ceph/cluster2-client.mirror.48074.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56203.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48074.log.gz 87.4% 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49526.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58333.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49410.log: gzip -5 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49410.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.72625.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42085.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58333.log.gz 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58581.log 2024-09-16T13:20:22.428 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72625.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60653.log 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58221.log 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58581.log: 87.5%/var/log/ceph/cluster2-client.mirror.60653.log: -- replaced with /var/log/ceph/cluster1-client.admin.42085.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68074.log 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58221.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60653.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58221.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74136.log 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68413.log 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68074.log: /var/log/ceph/cluster2-client.mirror.74136.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68074.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74136.log.gz 2024-09-16T13:20:22.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53155.log 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42944.log 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68413.log.gz 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55452.log 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53155.log.gz 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42944.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42944.log.gz 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63249.log 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73356.log 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55452.log: /var/log/ceph/cluster2-client.mirror.63249.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63249.log.gz 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55452.log.gz 2024-09-16T13:20:22.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51351.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69175.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73356.log: /var/log/ceph/cluster2-client.mirror.51351.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73356.log.gz 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57120.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51351.log.gz 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64079.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57120.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69175.log.gz 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46108.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57120.log.gz 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46457.log 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64079.log: /var/log/ceph/cluster2-client.mirror.46108.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64079.log.gz 2024-09-16T13:20:22.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43720.log 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46457.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46457.log.gz 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52164.log 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43720.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43720.log.gz 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67492.log 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.42148.log.gz 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52164.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52164.log.gz 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60431.log 2024-09-16T13:20:22.443 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43117.log 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46108.log.gz 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47589.log 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60431.log: /var/log/ceph/cluster1-client.admin.43117.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43117.log.gz 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62255.log 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47589.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67492.log.gz 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44044.log 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60431.log.gz/var/log/ceph/cluster2-client.mirror.62255.log: 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66810.log 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62255.log.gz 2024-09-16T13:20:22.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49368.log 2024-09-16T13:20:22.447 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44044.log: /var/log/ceph/cluster2-client.mirror.66810.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66810.log.gz 2024-09-16T13:20:22.447 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56343.log 2024-09-16T13:20:22.447 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49368.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49368.log.gz 2024-09-16T13:20:22.448 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62070.log 2024-09-16T13:20:22.448 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56343.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56343.log.gz 2024-09-16T13:20:22.448 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73213.log 2024-09-16T13:20:22.448 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62070.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62070.log.gz 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54966.log 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73213.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44044.log.gz 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55908.log 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73213.log.gz 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66521.log 2024-09-16T13:20:22.450 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55908.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54966.log.gz 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55908.log.gz 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57542.log 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62950.log 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57542.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66521.log.gz 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70503.log 2024-09-16T13:20:22.457 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62950.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57542.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62950.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61690.log 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.70503.log: -- /var/log/ceph/cluster1-client.admin.40541.log 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70503.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61690.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61690.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55227.log 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.38350.log 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.55227.log: -- replaced with /var/log/ceph/cluster1-client.admin.40541.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49221.log 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48559.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55227.log.gz 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.38350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68815.log 2024-09-16T13:20:22.458 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.49221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73126.log 2024-09-16T13:20:22.459 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68815.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68815.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.50940.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73126.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38350.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73126.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58581.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39289.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.50940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66542.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39306.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39289.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52185.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39306.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66542.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49221.log.gz 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48348.log 2024-09-16T13:20:22.467 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39306.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49788.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52185.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52185.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster2-client.mirror.48348.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52079.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49788.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48348.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74366.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40712.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52079.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49788.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46959.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74366.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52079.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40712.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40712.log.gz 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65508.log 2024-09-16T13:20:22.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67612.log 2024-09-16T13:20:22.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50067.log 2024-09-16T13:20:22.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67612.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67612.log.gz 2024-09-16T13:20:22.469 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58289.log 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50067.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65508.log.gz 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49605.log 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58289.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58289.log.gz 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60069.log 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49605.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49605.log.gz 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70143.log 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60069.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60069.log.gz 2024-09-16T13:20:22.472 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50146.log 2024-09-16T13:20:22.474 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70143.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70143.log.gz 2024-09-16T13:20:22.474 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65383.log 2024-09-16T13:20:22.477 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50146.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.50940.log.gz 2024-09-16T13:20:22.478 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62045.log 2024-09-16T13:20:22.480 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65383.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65383.log.gz 2024-09-16T13:20:22.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64638.log 2024-09-16T13:20:22.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62045.log.gz 2024-09-16T13:20:22.482 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65798.log 2024-09-16T13:20:22.483 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64638.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64638.log.gz 2024-09-16T13:20:22.483 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56915.log 2024-09-16T13:20:22.484 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65798.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65798.log.gz 2024-09-16T13:20:22.484 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56819.log 2024-09-16T13:20:22.485 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56915.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56915.log.gz 2024-09-16T13:20:22.485 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68635.log 2024-09-16T13:20:22.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56819.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56819.log.gz 2024-09-16T13:20:22.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56090.log 2024-09-16T13:20:22.489 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68635.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68635.log.gz 2024-09-16T13:20:22.489 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50730.log.gz 2024-09-16T13:20:22.489 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74445.log 2024-09-16T13:20:22.490 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58201.log 2024-09-16T13:20:22.490 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56090.log.gz 2024-09-16T13:20:22.490 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40411.log 2024-09-16T13:20:22.490 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58201.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74445.log.gz 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54327.log 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40411.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40411.log.gz 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58201.log.gz 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46337.log 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57610.log 2024-09-16T13:20:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46337.log: /var/log/ceph/cluster1-client.mirror.54327.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54327.log.gz 2024-09-16T13:20:22.495 INFO:teuthology.orchestra.run.smithi191.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46337.log.gz 2024-09-16T13:20:22.495 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72583.log 2024-09-16T13:20:22.495 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75609.log 2024-09-16T13:20:22.495 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57610.log: /var/log/ceph/cluster1-client.mirror.72583.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57610.log.gz 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.72583.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50434.log 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64751.log 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75609.log: /var/log/ceph/cluster2-client.mirror.50434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75609.log.gz 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50434.log.gz 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43956.log 2024-09-16T13:20:22.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73943.log 2024-09-16T13:20:22.497 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43956.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43956.log.gz 2024-09-16T13:20:22.497 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45157.log 2024-09-16T13:20:22.497 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73943.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64751.log.gz 2024-09-16T13:20:22.497 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66356.log 2024-09-16T13:20:22.498 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45157.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73943.log.gz 2024-09-16T13:20:22.498 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53718.log 2024-09-16T13:20:22.500 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66356.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45157.log.gz 2024-09-16T13:20:22.500 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66356.log.gz 2024-09-16T13:20:22.500 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58494.log 2024-09-16T13:20:22.500 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64662.log 2024-09-16T13:20:22.500 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58494.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58494.log.gz 2024-09-16T13:20:22.502 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51331.log 2024-09-16T13:20:22.502 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53718.log.gz 2024-09-16T13:20:22.502 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75425.log 2024-09-16T13:20:22.502 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51331.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64662.log.gz 2024-09-16T13:20:22.504 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62906.log 2024-09-16T13:20:22.504 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51331.log.gz/var/log/ceph/cluster1-client.admin.75425.log: 2024-09-16T13:20:22.504 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75425.log.gz 2024-09-16T13:20:22.505 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70458.log 2024-09-16T13:20:22.505 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49664.log 2024-09-16T13:20:22.505 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70458.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70458.log.gz 2024-09-16T13:20:22.505 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62906.log.gz 2024-09-16T13:20:22.505 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49929.log 2024-09-16T13:20:22.508 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57393.log 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49664.log: /var/log/ceph/cluster2-client.mirror.49929.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49929.log.gz 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59027.log 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7%/var/log/ceph/cluster1-client.mirror.57393.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49664.log.gz 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57393.log.gz 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51983.log 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46825.log 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59027.log: /var/log/ceph/cluster2-client.mirror.51983.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59027.log.gz 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51983.log.gz 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43573.log 2024-09-16T13:20:22.509 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48041.log 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43573.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46825.log.gz 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59529.log 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr: 90.0%/var/log/ceph/cluster2-client.mirror.48041.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43573.log.gz 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40644.log 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59529.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59529.log.gz 2024-09-16T13:20:22.512 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42828.log 2024-09-16T13:20:22.513 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40644.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40644.log.gz 2024-09-16T13:20:22.514 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73797.log 2024-09-16T13:20:22.515 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42828.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42828.log.gz 2024-09-16T13:20:22.515 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48091.log 2024-09-16T13:20:22.516 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73797.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73797.log.gz 2024-09-16T13:20:22.517 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70165.log 2024-09-16T13:20:22.518 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48091.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48091.log.gz 2024-09-16T13:20:22.518 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46537.log 2024-09-16T13:20:22.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70165.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70165.log.gz 2024-09-16T13:20:22.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.48949.log 2024-09-16T13:20:22.521 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46537.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46537.log.gz 2024-09-16T13:20:22.521 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74963.log 2024-09-16T13:20:22.532 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.48949.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49627.log.gz 2024-09-16T13:20:22.533 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59804.log 2024-09-16T13:20:22.534 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74963.log: 75.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74963.log.gz 2024-09-16T13:20:22.538 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58955.log 2024-09-16T13:20:22.539 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59804.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59804.log.gz 2024-09-16T13:20:22.539 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60733.log 2024-09-16T13:20:22.540 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58955.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58955.log.gz 2024-09-16T13:20:22.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64213.log 2024-09-16T13:20:22.542 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60733.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60733.log.gz 2024-09-16T13:20:22.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50861.log 2024-09-16T13:20:22.543 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64213.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64213.log.gz 2024-09-16T13:20:22.543 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53179.log 2024-09-16T13:20:22.545 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50861.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50861.log.gz 2024-09-16T13:20:22.545 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65218.log 2024-09-16T13:20:22.546 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53179.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53179.log.gz 2024-09-16T13:20:22.547 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55773.log 2024-09-16T13:20:22.548 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65218.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65218.log.gz 2024-09-16T13:20:22.548 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47070.log 2024-09-16T13:20:22.548 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55773.log.gz 2024-09-16T13:20:22.549 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73446.log 2024-09-16T13:20:22.559 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-16T13:20:22.565 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73446.log: 79.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70525.log 2024-09-16T13:20:22.565 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73446.log.gz 2024-09-16T13:20:22.573 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-16T13:20:22.579 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66149.log 2024-09-16T13:20:22.579 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70525.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70525.log.gz 2024-09-16T13:20:22.592 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55520.log 2024-09-16T13:20:22.592 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66149.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66149.log.gz 2024-09-16T13:20:22.601 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59184.log 2024-09-16T13:20:22.601 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55520.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55520.log.gz 2024-09-16T13:20:22.619 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65570.log 2024-09-16T13:20:22.619 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59184.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59184.log.gz 2024-09-16T13:20:22.628 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66955.log 2024-09-16T13:20:22.629 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65570.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65570.log.gz 2024-09-16T13:20:22.641 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62279.log 2024-09-16T13:20:22.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66955.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66955.log.gz 2024-09-16T13:20:22.660 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66790.log 2024-09-16T13:20:22.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62279.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62279.log.gz 2024-09-16T13:20:22.678 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33179.log 2024-09-16T13:20:22.678 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66790.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66790.log.gz 2024-09-16T13:20:22.685 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73148.log.gz 2024-09-16T13:20:22.700 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37963.log 2024-09-16T13:20:22.701 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.33179.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33179.log.gz 2024-09-16T13:20:22.713 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49946.log 2024-09-16T13:20:22.714 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37963.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37963.log.gz 2024-09-16T13:20:22.722 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37311.log 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49946.log: 89.7% 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.48949.log.gz 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49946.log.gz 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46680.log 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37311.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37311.log.gz 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63739.log 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46680.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46680.log.gz 2024-09-16T13:20:22.730 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74884.log 2024-09-16T13:20:22.731 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63739.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63739.log.gz 2024-09-16T13:20:22.732 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51641.log 2024-09-16T13:20:22.732 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74884.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74884.log.gz 2024-09-16T13:20:22.733 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61421.log 2024-09-16T13:20:22.734 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51641.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51641.log.gz 2024-09-16T13:20:22.734 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57482.log 2024-09-16T13:20:22.735 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61421.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48041.log.gz 2024-09-16T13:20:22.736 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61421.log.gz 2024-09-16T13:20:22.736 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50249.log 2024-09-16T13:20:22.736 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54641.log 2024-09-16T13:20:22.736 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57482.log: /var/log/ceph/cluster2-client.mirror.50249.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50249.log.gz 2024-09-16T13:20:22.737 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73820.log 2024-09-16T13:20:22.737 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54641.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57482.log.gz 2024-09-16T13:20:22.748 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50067.log.gz 2024-09-16T13:20:22.748 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54641.log.gz 2024-09-16T13:20:22.748 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68770.log 2024-09-16T13:20:22.748 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61441.log 2024-09-16T13:20:22.748 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68770.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73820.log.gz 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53940.log 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68770.log.gz 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.61441.log: --verbose -- /var/log/ceph/cluster2-client.mirror.50471.log 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53940.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61441.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37079.log 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53940.log.gz 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62118.log 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50471.log: /var/log/ceph/cluster2-client.admin.37079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45569.log 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37079.log.gz 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50471.log.gz 2024-09-16T13:20:22.749 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53629.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64594.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62118.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67100.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64594.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53629.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51423.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8%/var/log/ceph/cluster2-client.mirror.67100.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64594.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63992.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67100.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70323.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63992.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63992.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51423.log.gz 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55613.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75306.log 2024-09-16T13:20:22.750 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70323.log.gz 2024-09-16T13:20:22.751 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47909.log 2024-09-16T13:20:22.751 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75306.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45569.log.gz 2024-09-16T13:20:22.751 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75306.log.gz 2024-09-16T13:20:22.751 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69468.log 2024-09-16T13:20:22.751 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59616.log 2024-09-16T13:20:22.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69468.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69468.log.gz 2024-09-16T13:20:22.762 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55613.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47909.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65942.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.59616.log: -- /var/log/ceph/cluster1-client.mirror.52633.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44287.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52633.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59616.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52633.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64686.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.44287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43347.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44287.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64686.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65942.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64686.log.gz 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51799.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69220.log 2024-09-16T13:20:22.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66831.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43347.log.gz/var/log/ceph/cluster2-client.mirror.51799.log: 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69220.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69220.log.gz 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44407.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59852.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.44407.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66831.log.gz 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66211.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44407.log.gz/var/log/ceph/cluster1-client.mirror.59852.log: 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55323.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59852.log.gz 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44383.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66211.log: /var/log/ceph/cluster1-client.mirror.55323.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55323.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66211.log.gz 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53988.log 2024-09-16T13:20:22.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.44383.log: -- /var/log/ceph/cluster1-client.mirror.70707.log 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44383.log.gz 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53988.log: 87.4%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.53988.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46297.log 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.59691.log 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46297.log: /var/log/ceph/cluster1-client.mirror.70707.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70707.log.gz 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42111.log 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46297.log.gz 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.59691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38187.log 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42111.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42111.log.gz 2024-09-16T13:20:22.765 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72669.log 2024-09-16T13:20:22.769 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.38187.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38187.log.gz 2024-09-16T13:20:22.769 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66645.log 2024-09-16T13:20:22.770 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72669.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72669.log.gz 2024-09-16T13:20:22.770 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53581.log 2024-09-16T13:20:22.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66645.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66645.log.gz 2024-09-16T13:20:22.771 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51225.log 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53581.log: 87.3% 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.59691.log.gz 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53581.log.gz 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51888.log 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67743.log 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51888.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51888.log.gz 2024-09-16T13:20:22.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59208.log 2024-09-16T13:20:22.774 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67743.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67743.log.gz 2024-09-16T13:20:22.774 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73640.log 2024-09-16T13:20:22.777 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59208.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59208.log.gz 2024-09-16T13:20:22.777 INFO:teuthology.orchestra.run.smithi191.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51225.log.gz 2024-09-16T13:20:22.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63607.log 2024-09-16T13:20:22.778 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54882.log 2024-09-16T13:20:22.778 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73640.log.gz 2024-09-16T13:20:22.778 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76030.log 2024-09-16T13:20:22.778 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54882.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63607.log.gz 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54882.log.gz 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40363.log 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64831.log 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40363.log: /var/log/ceph/cluster2-client.admin.76030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76030.log.gz 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40363.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67182.log 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.780 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59481.log 2024-09-16T13:20:22.782 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67182.log: 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64831.log.gz 2024-09-16T13:20:22.782 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46257.log 2024-09-16T13:20:22.782 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59481.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59481.log.gz 2024-09-16T13:20:22.783 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59553.log 2024-09-16T13:20:22.784 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46257.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46257.log.gz 2024-09-16T13:20:22.784 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67182.log.gz 2024-09-16T13:20:22.784 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66025.log 2024-09-16T13:20:22.784 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68115.log 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59553.log: /var/log/ceph/cluster2-client.mirror.66025.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59553.log.gz 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38738.log 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66025.log.gz 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38297.log 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68115.log: /var/log/ceph/cluster1-client.mirror.38738.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38738.log.gz 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68115.log.gz 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41865.log 2024-09-16T13:20:22.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.38297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50802.log 2024-09-16T13:20:22.791 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41865.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51799.log.gz 2024-09-16T13:20:22.791 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65632.log 2024-09-16T13:20:22.792 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50802.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50802.log.gz 2024-09-16T13:20:22.792 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41865.log.gz 2024-09-16T13:20:22.793 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49684.log 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65632.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65632.log.gz 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33000.log 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65405.log 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49684.log.gz 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.33000.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33000.log.gz 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41453.log 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40664.log 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65405.log.gz/var/log/ceph/cluster2-client.mirror.41453.log: 2024-09-16T13:20:22.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54480.log 2024-09-16T13:20:22.799 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40664.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40664.log.gz 2024-09-16T13:20:22.799 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66893.log 2024-09-16T13:20:22.800 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54480.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54480.log.gz 2024-09-16T13:20:22.801 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66252.log 2024-09-16T13:20:22.801 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66893.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66893.log.gz 2024-09-16T13:20:22.804 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54375.log 2024-09-16T13:20:22.804 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41453.log.gz 2024-09-16T13:20:22.804 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51145.log 2024-09-16T13:20:22.804 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54375.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54375.log.gz 2024-09-16T13:20:22.804 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45334.log 2024-09-16T13:20:22.805 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51145.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51145.log.gz 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66252.log.gz 2024-09-16T13:20:22.805 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.805 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58694.log 2024-09-16T13:20:22.806 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39199.log 2024-09-16T13:20:22.808 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45334.log: /var/log/ceph/cluster1-client.mirror.58694.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58694.log.gz 2024-09-16T13:20:22.808 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45201.log 2024-09-16T13:20:22.808 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39199.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39199.log.gz 2024-09-16T13:20:22.808 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59092.log 2024-09-16T13:20:22.809 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45201.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45201.log.gz 2024-09-16T13:20:22.809 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44064.log 2024-09-16T13:20:22.811 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59092.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59092.log.gz 2024-09-16T13:20:22.811 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49427.log 2024-09-16T13:20:22.813 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44064.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38297.log.gz 2024-09-16T13:20:22.813 INFO:teuthology.orchestra.run.smithi191.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45334.log.gz 2024-09-16T13:20:22.813 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41365.log 2024-09-16T13:20:22.813 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73319.log 2024-09-16T13:20:22.814 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41365.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49427.log.gz 2024-09-16T13:20:22.814 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41365.log.gz 2024-09-16T13:20:22.814 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42383.log 2024-09-16T13:20:22.814 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63808.log 2024-09-16T13:20:22.815 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73319.log: /var/log/ceph/cluster1-client.mirror.42383.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42383.log.gz 2024-09-16T13:20:22.815 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50087.log 2024-09-16T13:20:22.816 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63808.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44064.log.gz 2024-09-16T13:20:22.818 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73319.log.gz 2024-09-16T13:20:22.818 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67867.log 2024-09-16T13:20:22.818 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63808.log.gz 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68218.log 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50087.log: /var/log/ceph/cluster2-client.mirror.67867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43595.log 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50087.log.gz 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65081.log 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67867.log.gz 2024-09-16T13:20:22.819 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43144.log 2024-09-16T13:20:22.820 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.65081.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65081.log.gz 2024-09-16T13:20:22.820 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68218.log.gz 2024-09-16T13:20:22.820 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61513.log 2024-09-16T13:20:22.821 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40192.log 2024-09-16T13:20:22.821 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.43144.log: /var/log/ceph/cluster1-client.mirror.61513.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43144.log.gz 2024-09-16T13:20:22.822 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61513.log.gz 2024-09-16T13:20:22.822 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46745.log 2024-09-16T13:20:22.822 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64614.log 2024-09-16T13:20:22.823 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40192.log: /var/log/ceph/cluster2-client.mirror.46745.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40192.log.gz 2024-09-16T13:20:22.823 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70368.log 2024-09-16T13:20:22.829 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46745.log.gz/var/log/ceph/cluster2-client.mirror.64614.log: 2024-09-16T13:20:22.829 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57441.log 2024-09-16T13:20:22.829 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64614.log.gz 2024-09-16T13:20:22.829 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70368.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70368.log.gz 2024-09-16T13:20:22.829 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60560.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56247.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60560.log: /var/log/ceph/cluster1-client.mirror.57441.log: 87.4% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57441.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60560.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65963.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.56247.log: -- /var/log/ceph/cluster2-client.mirror.73278.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56247.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57144.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73278.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65963.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73278.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64390.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72086.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57144.log: /var/log/ceph/cluster2-client.mirror.64390.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57144.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64390.log.gz 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72372.log 2024-09-16T13:20:22.830 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42267.log 2024-09-16T13:20:22.832 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72086.log: /var/log/ceph/cluster1-client.mirror.72372.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72372.log.gz 2024-09-16T13:20:22.832 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49830.log 2024-09-16T13:20:22.832 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42267.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42267.log.gz 2024-09-16T13:20:22.832 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45107.log 2024-09-16T13:20:22.834 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49830.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47589.log.gz 2024-09-16T13:20:22.835 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68950.log 2024-09-16T13:20:22.836 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45107.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45107.log.gz 2024-09-16T13:20:22.836 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46577.log 2024-09-16T13:20:22.838 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68950.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68950.log.gz 2024-09-16T13:20:22.838 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54303.log 2024-09-16T13:20:22.838 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46577.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46577.log.gz 2024-09-16T13:20:22.839 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62476.log 2024-09-16T13:20:22.839 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54303.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54303.log.gz 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36862.log 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62476.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50146.log.gz 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62476.log.gz 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62864.log 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57233.log 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.36862.log.gz 2024-09-16T13:20:22.842 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43629.log 2024-09-16T13:20:22.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57233.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62864.log.gz 2024-09-16T13:20:22.843 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75203.log 2024-09-16T13:20:22.844 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.43629.log: 92.4% 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43629.log.gz 2024-09-16T13:20:22.844 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57233.log.gz 2024-09-16T13:20:22.844 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58853.log 2024-09-16T13:20:22.844 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67079.log 2024-09-16T13:20:22.845 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75203.log: /var/log/ceph/cluster2-client.mirror.58853.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58853.log.gz 2024-09-16T13:20:22.846 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40103.log 2024-09-16T13:20:22.847 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67079.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67079.log.gz 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41473.log 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.75203.log.gz 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40103.log: 74.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40103.log.gz 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68012.log 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68545.log 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68012.log: /var/log/ceph/cluster1-client.mirror.41473.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68012.log.gz 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62572.log 2024-09-16T13:20:22.850 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68545.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68545.log.gz 2024-09-16T13:20:22.853 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36837.log 2024-09-16T13:20:22.853 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62572.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62572.log.gz 2024-09-16T13:20:22.853 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41269.log 2024-09-16T13:20:22.853 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36837.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36837.log.gz 2024-09-16T13:20:22.853 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55797.log 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41269.log: 93.5% 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41269.log.gz 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41473.log.gz 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69445.log 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45085.log 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55797.log.gz 2024-09-16T13:20:22.856 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69445.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69445.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52609.log 2024-09-16T13:20:22.857 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:22.857 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75256.log 2024-09-16T13:20:22.857 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52609.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52609.log.gz 2024-09-16T13:20:22.858 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42407.log 2024-09-16T13:20:22.858 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75256.log.gz 2024-09-16T13:20:22.858 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56947.log 2024-09-16T13:20:22.858 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42407.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42407.log.gz 2024-09-16T13:20:22.859 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58243.log 2024-09-16T13:20:22.859 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56947.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56947.log.gz 2024-09-16T13:20:22.860 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55251.log 2024-09-16T13:20:22.861 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58243.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58243.log.gz 2024-09-16T13:20:22.861 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45085.log.gz 2024-09-16T13:20:22.861 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52786.log 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.55251.log: -- /var/log/ceph/cluster2-client.mirror.73864.log 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55251.log.gz 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59780.log 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73864.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73864.log.gz 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52786.log.gz 2024-09-16T13:20:22.866 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75183.log 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53964.log 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59780.log.gz 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58544.log 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53964.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75183.log.gz 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45042.log 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53964.log.gz 2024-09-16T13:20:22.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42968.log 2024-09-16T13:20:22.868 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45042.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58544.log.gz 2024-09-16T13:20:22.868 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60890.log 2024-09-16T13:20:22.868 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42968.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42968.log.gz 2024-09-16T13:20:22.868 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65674.log 2024-09-16T13:20:22.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60890.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60890.log.gz 2024-09-16T13:20:22.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62524.log 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65674.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65674.log.gz 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45042.log.gz 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40934.log 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68389.log 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62524.log: /var/log/ceph/cluster1-client.admin.40934.log: 87.4% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40934.log.gz 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62524.log.gz 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65591.log 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70255.log 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68389.log.gz 2024-09-16T13:20:22.876 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40688.log 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70255.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70255.log.gz 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65591.log.gz 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48825.log 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40688.log.gz 2024-09-16T13:20:22.877 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57213.log 2024-09-16T13:20:22.879 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48825.log: 92.2% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-16T13:20:22.880 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40456.log 2024-09-16T13:20:22.881 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57213.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57213.log.gz 2024-09-16T13:20:22.882 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40912.log 2024-09-16T13:20:22.887 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40456.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40456.log.gz 2024-09-16T13:20:22.888 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66666.log 2024-09-16T13:20:22.901 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39250.log 2024-09-16T13:20:22.902 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66666.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66666.log.gz 2024-09-16T13:20:22.908 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47070.log.gz 2024-09-16T13:20:22.909 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47530.log 2024-09-16T13:20:22.909 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39250.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39250.log.gz 2024-09-16T13:20:22.911 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44001.log 2024-09-16T13:20:22.911 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47530.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47530.log.gz 2024-09-16T13:20:22.911 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59114.log 2024-09-16T13:20:22.916 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44001.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44001.log.gz 2024-09-16T13:20:22.917 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57723.log 2024-09-16T13:20:22.919 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59114.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59114.log.gz 2024-09-16T13:20:22.919 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43032.log 2024-09-16T13:20:22.921 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57723.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57723.log.gz 2024-09-16T13:20:22.921 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53465.log 2024-09-16T13:20:22.926 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43032.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43032.log.gz 2024-09-16T13:20:22.927 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62822.log 2024-09-16T13:20:22.930 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53465.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53465.log.gz 2024-09-16T13:20:22.930 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47348.log 2024-09-16T13:20:22.932 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62822.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62822.log.gz 2024-09-16T13:20:22.932 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60512.log 2024-09-16T13:20:22.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68613.log 2024-09-16T13:20:22.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60512.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60512.log.gz 2024-09-16T13:20:22.957 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75567.log 2024-09-16T13:20:22.958 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68613.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68613.log.gz 2024-09-16T13:20:22.958 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39233.log 2024-09-16T13:20:22.958 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75567.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75567.log.gz 2024-09-16T13:20:22.963 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46959.log.gz 2024-09-16T13:20:22.970 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41661.log 2024-09-16T13:20:22.970 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39233.log: 56.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39233.log.gz 2024-09-16T13:20:22.970 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70801.log 2024-09-16T13:20:22.971 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41661.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41661.log.gz 2024-09-16T13:20:22.971 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.73356.log 2024-09-16T13:20:22.984 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67038.log 2024-09-16T13:20:22.987 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.73356.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.73356.log.gz 2024-09-16T13:20:22.991 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65467.log 2024-09-16T13:20:23.004 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42471.log 2024-09-16T13:20:23.004 INFO:teuthology.orchestra.run.smithi191.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67038.log.gz 2024-09-16T13:20:23.004 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65467.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65467.log.gz 2024-09-16T13:20:23.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43682.log 2024-09-16T13:20:23.023 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42471.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42471.log.gz 2024-09-16T13:20:23.031 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47999.log 2024-09-16T13:20:23.032 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43682.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43682.log.gz 2024-09-16T13:20:23.043 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64462.log 2024-09-16T13:20:23.045 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47999.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47999.log.gz 2024-09-16T13:20:23.051 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52541.log 2024-09-16T13:20:23.051 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64462.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64462.log.gz 2024-09-16T13:20:23.060 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60492.log 2024-09-16T13:20:23.066 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52541.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52541.log.gz 2024-09-16T13:20:23.077 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50881.log 2024-09-16T13:20:23.079 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60492.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60492.log.gz 2024-09-16T13:20:23.086 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69760.log 2024-09-16T13:20:23.097 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52121.log 2024-09-16T13:20:23.097 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69760.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69760.log.gz 2024-09-16T13:20:23.099 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50881.log.gz 2024-09-16T13:20:23.105 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52300.log 2024-09-16T13:20:23.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52121.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43595.log.gz 2024-09-16T13:20:23.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47646.log 2024-09-16T13:20:23.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52300.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52300.log.gz 2024-09-16T13:20:23.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64302.log 2024-09-16T13:20:23.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47646.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47646.log.gz 2024-09-16T13:20:23.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66686.log 2024-09-16T13:20:23.147 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64302.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64302.log.gz 2024-09-16T13:20:23.147 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61212.log 2024-09-16T13:20:23.150 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66686.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66686.log.gz 2024-09-16T13:20:23.150 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59963.log 2024-09-16T13:20:23.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61212.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61212.log.gz 2024-09-16T13:20:23.165 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44977.log 2024-09-16T13:20:23.165 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59963.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59963.log.gz 2024-09-16T13:20:23.177 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40912.log.gz 2024-09-16T13:20:23.178 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49055.log 2024-09-16T13:20:23.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44977.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44977.log.gz 2024-09-16T13:20:23.179 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45177.log 2024-09-16T13:20:23.180 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49055.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49055.log.gz 2024-09-16T13:20:23.181 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55127.log 2024-09-16T13:20:23.182 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45177.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45177.log.gz 2024-09-16T13:20:23.182 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54813.log 2024-09-16T13:20:23.184 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55127.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55127.log.gz 2024-09-16T13:20:23.184 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48291.log 2024-09-16T13:20:23.186 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54813.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54813.log.gz 2024-09-16T13:20:23.186 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.64920.log 2024-09-16T13:20:23.198 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48291.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73047.log 2024-09-16T13:20:23.217 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.3.64920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39326.log 2024-09-16T13:20:23.231 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73047.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73047.log.gz 2024-09-16T13:20:23.235 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65363.log 2024-09-16T13:20:23.235 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39326.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39326.log.gz 2024-09-16T13:20:23.243 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46900.log 2024-09-16T13:20:23.253 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61714.log 2024-09-16T13:20:23.253 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46900.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46900.log.gz 2024-09-16T13:20:23.257 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65363.log.gz 2024-09-16T13:20:23.260 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49830.log.gz 2024-09-16T13:20:23.261 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62777.log 2024-09-16T13:20:23.261 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61714.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61714.log.gz 2024-09-16T13:20:23.262 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.73550.log 2024-09-16T13:20:23.263 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62777.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62777.log.gz 2024-09-16T13:20:23.263 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60342.log 2024-09-16T13:20:23.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.73550.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.73550.log.gz 2024-09-16T13:20:23.267 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70548.log 2024-09-16T13:20:23.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60342.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60342.log.gz 2024-09-16T13:20:23.269 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64486.log 2024-09-16T13:20:23.270 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70548.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70548.log.gz 2024-09-16T13:20:23.270 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72982.log 2024-09-16T13:20:23.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64486.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64486.log.gz 2024-09-16T13:20:23.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55970.log 2024-09-16T13:20:23.273 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72982.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72982.log.gz 2024-09-16T13:20:23.273 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74115.log 2024-09-16T13:20:23.274 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55970.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55970.log.gz 2024-09-16T13:20:23.274 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73963.log 2024-09-16T13:20:23.275 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74115.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74115.log.gz 2024-09-16T13:20:23.275 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60536.log 2024-09-16T13:20:23.276 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73963.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73963.log.gz 2024-09-16T13:20:23.277 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49850.log 2024-09-16T13:20:23.277 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60536.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60536.log.gz 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42057.log 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47348.log.gz 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49850.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49850.log.gz 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55368.log 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75651.log 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42057.log.gz 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62548.log 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75651.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75651.log.gz 2024-09-16T13:20:23.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63944.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62548.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62548.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55368.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57168.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63944.log: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49771.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72086.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63944.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37345.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49771.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57168.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49771.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53328.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.37345.log: -- /var/log/ceph/cluster2-client.mirror.48133.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37345.log.gz 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74428.log 2024-09-16T13:20:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53328.log.gz 2024-09-16T13:20:23.288 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50671.log 2024-09-16T13:20:23.288 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74428.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74428.log.gz 2024-09-16T13:20:23.289 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56651.log 2024-09-16T13:20:23.289 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50671.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50671.log.gz 2024-09-16T13:20:23.289 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63787.log 2024-09-16T13:20:23.289 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56651.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56651.log.gz 2024-09-16T13:20:23.289 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54741.log 2024-09-16T13:20:23.290 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63787.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63787.log.gz 2024-09-16T13:20:23.290 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45022.log 2024-09-16T13:20:23.290 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54741.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54741.log.gz 2024-09-16T13:20:23.291 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43860.log 2024-09-16T13:20:23.297 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45022.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45022.log.gz 2024-09-16T13:20:23.297 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54986.log 2024-09-16T13:20:23.297 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43860.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43860.log.gz 2024-09-16T13:20:23.298 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49097.log 2024-09-16T13:20:23.298 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54986.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54986.log.gz 2024-09-16T13:20:23.299 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73843.log 2024-09-16T13:20:23.304 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49097.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.64920.log.gz 2024-09-16T13:20:23.306 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67991.log 2024-09-16T13:20:23.306 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73843.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73843.log.gz 2024-09-16T13:20:23.306 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68094.log 2024-09-16T13:20:23.308 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67991.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67991.log.gz 2024-09-16T13:20:23.308 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73234.log 2024-09-16T13:20:23.310 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68094.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68094.log.gz 2024-09-16T13:20:23.311 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67162.log 2024-09-16T13:20:23.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73234.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73234.log.gz 2024-09-16T13:20:23.312 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50226.log 2024-09-16T13:20:23.313 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67162.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67162.log.gz 2024-09-16T13:20:23.313 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72648.log 2024-09-16T13:20:23.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42428.log 2024-09-16T13:20:23.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72648.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72648.log.gz 2024-09-16T13:20:23.331 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66583.log 2024-09-16T13:20:23.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56448.log 2024-09-16T13:20:23.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66583.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66583.log.gz 2024-09-16T13:20:23.348 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42428.log.gz 2024-09-16T13:20:23.352 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48825.log.gz 2024-09-16T13:20:23.352 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64811.log 2024-09-16T13:20:23.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56448.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56448.log.gz 2024-09-16T13:20:23.353 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53400.log 2024-09-16T13:20:23.355 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64811.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64811.log.gz 2024-09-16T13:20:23.355 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49548.log 2024-09-16T13:20:23.356 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53400.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53400.log.gz 2024-09-16T13:20:23.356 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60021.log 2024-09-16T13:20:23.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67950.log 2024-09-16T13:20:23.369 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60021.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60021.log.gz 2024-09-16T13:20:23.386 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52744.log 2024-09-16T13:20:23.386 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67950.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67950.log.gz 2024-09-16T13:20:23.389 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70801.log.gz 2024-09-16T13:20:23.392 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52121.log.gz 2024-09-16T13:20:23.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67451.log 2024-09-16T13:20:23.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43327.log 2024-09-16T13:20:23.401 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52744.log: /var/log/ceph/cluster2-client.mirror.67451.log: 89.7% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67451.log.gz 2024-09-16T13:20:23.401 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52744.log.gz 2024-09-16T13:20:23.401 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.43207.log 2024-09-16T13:20:23.401 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45592.log 2024-09-16T13:20:23.402 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.43207.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43327.log.gz 2024-09-16T13:20:23.402 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65715.log 2024-09-16T13:20:23.403 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45592.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.43207.log.gz 2024-09-16T13:20:23.404 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39999.log 2024-09-16T13:20:23.405 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65715.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65715.log.gz 2024-09-16T13:20:23.405 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72416.log 2024-09-16T13:20:23.408 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39999.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45592.log.gz 2024-09-16T13:20:23.408 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53131.log 2024-09-16T13:20:23.410 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72416.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72416.log.gz 2024-09-16T13:20:23.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52440.log 2024-09-16T13:20:23.410 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53131.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53131.log.gz 2024-09-16T13:20:23.411 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51074.log 2024-09-16T13:20:23.411 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52440.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52440.log.gz 2024-09-16T13:20:23.413 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74214.log 2024-09-16T13:20:23.413 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51074.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51074.log.gz 2024-09-16T13:20:23.413 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40103.log 2024-09-16T13:20:23.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74214.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74214.log.gz 2024-09-16T13:20:23.415 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72790.log 2024-09-16T13:20:23.418 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.40103.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40103.log.gz 2024-09-16T13:20:23.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50513.log 2024-09-16T13:20:23.432 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57859.log 2024-09-16T13:20:23.432 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50513.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50513.log.gz 2024-09-16T13:20:23.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54438.log 2024-09-16T13:20:23.453 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46128.log 2024-09-16T13:20:23.453 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57859.log.gz 2024-09-16T13:20:23.453 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54438.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54438.log.gz 2024-09-16T13:20:23.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61807.log 2024-09-16T13:20:23.471 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46128.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46128.log.gz 2024-09-16T13:20:23.485 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66294.log 2024-09-16T13:20:23.486 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61807.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61807.log.gz 2024-09-16T13:20:23.492 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38047.log 2024-09-16T13:20:23.492 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66294.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66294.log.gz 2024-09-16T13:20:23.501 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75861.log 2024-09-16T13:20:23.501 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.38047.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38047.log.gz 2024-09-16T13:20:23.504 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48133.log.gz 2024-09-16T13:20:23.504 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61349.log 2024-09-16T13:20:23.504 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75861.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75861.log.gz 2024-09-16T13:20:23.504 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60632.log 2024-09-16T13:20:23.506 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61349.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61349.log.gz 2024-09-16T13:20:23.506 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73905.log 2024-09-16T13:20:23.508 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60632.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60632.log.gz 2024-09-16T13:20:23.508 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73005.log 2024-09-16T13:20:23.510 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73905.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73905.log.gz 2024-09-16T13:20:23.510 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52227.log 2024-09-16T13:20:23.513 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73005.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73005.log.gz 2024-09-16T13:20:23.513 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56111.log 2024-09-16T13:20:23.513 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52227.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52227.log.gz 2024-09-16T13:20:23.513 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59280.log 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56111.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49097.log.gz 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56111.log.gz 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51545.log 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54683.log 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59280.log.gz 2024-09-16T13:20:23.519 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40961.log 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr: 79.8%/var/log/ceph/cluster2-client.mirror.54683.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51545.log.gz 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46237.log 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54683.log.gz 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44905.log 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40961.log.gz 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46237.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46237.log.gz 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58132.log 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63968.log 2024-09-16T13:20:23.520 INFO:teuthology.orchestra.run.smithi191.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44905.log.gz 2024-09-16T13:20:23.526 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67327.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.63968.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58132.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56968.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63968.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63321.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56968.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67327.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52561.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63321.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63321.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74322.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56968.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37204.log 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.74322.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52561.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74322.log.gz 2024-09-16T13:20:23.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45913.log 2024-09-16T13:20:23.528 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37204.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37204.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65653.log 2024-09-16T13:20:23.528 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:23.528 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66976.log 2024-09-16T13:20:23.528 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45913.log: /var/log/ceph/cluster2-client.mirror.65653.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65653.log.gz 2024-09-16T13:20:23.528 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57321.log 2024-09-16T13:20:23.530 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66976.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66976.log.gz 2024-09-16T13:20:23.531 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55866.log 2024-09-16T13:20:23.531 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57321.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57321.log.gz 2024-09-16T13:20:23.531 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68342.log 2024-09-16T13:20:23.531 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55866.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45913.log.gz 2024-09-16T13:20:23.532 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55866.log.gz 2024-09-16T13:20:23.532 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62235.log 2024-09-16T13:20:23.532 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59941.log 2024-09-16T13:20:23.533 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62235.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62235.log.gz 2024-09-16T13:20:23.533 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40520.log 2024-09-16T13:20:23.534 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59941.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68342.log.gz 2024-09-16T13:20:23.534 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65694.log 2024-09-16T13:20:23.536 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40520.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49548.log.gz 2024-09-16T13:20:23.536 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59941.log.gz 2024-09-16T13:20:23.536 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59876.log 2024-09-16T13:20:23.537 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.65694.log: -- /var/log/ceph/cluster2-client.mirror.62414.log 2024-09-16T13:20:23.537 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59876.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59876.log.gz 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57682.log 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62414.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65694.log.gz 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58790.log 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57682.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62414.log.gz 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39165.log 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57682.log.gz 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51054.log 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39165.log: 52.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39165.log.gz 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58790.log.gz 2024-09-16T13:20:23.541 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:23.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72457.log 2024-09-16T13:20:23.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73463.log 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51054.log: /var/log/ceph/cluster2-client.mirror.72457.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50226.log.gz 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51054.log.gz 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50592.log 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49867.log 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50592.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50592.log.gz 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73463.log.gz 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47668.log 2024-09-16T13:20:23.548 INFO:teuthology.orchestra.run.smithi191.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72457.log.gz 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49184.log 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49867.log: gzip -5/var/log/ceph/cluster2-client.mirror.47668.log: --verbose -- /var/log/ceph/cluster2-client.admin.75449.log 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.49184.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49867.log.gz 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67805.log 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75449.log.gz 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65426.log 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67805.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67805.log.gz 2024-09-16T13:20:23.549 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44503.log 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65426.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65426.log.gz 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56492.log 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44503.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44503.log.gz 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70210.log 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56492.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56492.log.gz 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43531.log 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70210.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70210.log.gz 2024-09-16T13:20:23.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54721.log 2024-09-16T13:20:23.559 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43531.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43531.log.gz 2024-09-16T13:20:23.559 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41709.log 2024-09-16T13:20:23.561 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54721.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54721.log.gz 2024-09-16T13:20:23.561 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51683.log 2024-09-16T13:20:23.562 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41709.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41709.log.gz 2024-09-16T13:20:23.562 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41293.log 2024-09-16T13:20:23.573 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46065.log 2024-09-16T13:20:23.574 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41293.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41293.log.gz 2024-09-16T13:20:23.587 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41538.log 2024-09-16T13:20:23.592 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46065.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46065.log.gz 2024-09-16T13:20:23.594 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64237.log 2024-09-16T13:20:23.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.41538.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41538.log.gz 2024-09-16T13:20:23.607 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66190.log 2024-09-16T13:20:23.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64237.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64237.log.gz 2024-09-16T13:20:23.620 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60001.log 2024-09-16T13:20:23.622 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66190.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66190.log.gz 2024-09-16T13:20:23.631 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48370.log 2024-09-16T13:20:23.633 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60001.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60001.log.gz 2024-09-16T13:20:23.636 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48291.log.gz 2024-09-16T13:20:23.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46658.log 2024-09-16T13:20:23.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48370.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72790.log.gz 2024-09-16T13:20:23.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49489.log 2024-09-16T13:20:23.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46658.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46658.log.gz 2024-09-16T13:20:23.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38670.log 2024-09-16T13:20:23.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49489.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49489.log.gz 2024-09-16T13:20:23.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74805.log 2024-09-16T13:20:23.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38670.log: 66.7% -- replaced with /var/log/ceph/cluster1-client.admin.38670.log.gz 2024-09-16T13:20:23.644 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60695.log 2024-09-16T13:20:23.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74805.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74805.log.gz 2024-09-16T13:20:23.646 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63828.log 2024-09-16T13:20:23.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60695.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60695.log.gz 2024-09-16T13:20:23.646 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57994.log 2024-09-16T13:20:23.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63828.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63828.log.gz 2024-09-16T13:20:23.648 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59367.log 2024-09-16T13:20:23.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57994.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57994.log.gz 2024-09-16T13:20:23.649 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38271.log 2024-09-16T13:20:23.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59367.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59367.log.gz 2024-09-16T13:20:23.652 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53445.log 2024-09-16T13:20:23.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38271.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38271.log.gz 2024-09-16T13:20:23.652 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67224.log 2024-09-16T13:20:23.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53445.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53445.log.gz 2024-09-16T13:20:23.654 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66314.log 2024-09-16T13:20:23.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67224.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67224.log.gz 2024-09-16T13:20:23.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69670.log 2024-09-16T13:20:23.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66314.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66314.log.gz 2024-09-16T13:20:23.658 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46867.log 2024-09-16T13:20:23.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69670.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69670.log.gz 2024-09-16T13:20:23.660 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60777.log 2024-09-16T13:20:23.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63671.log 2024-09-16T13:20:23.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60777.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60777.log.gz 2024-09-16T13:20:23.686 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58766.log 2024-09-16T13:20:23.688 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63671.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63671.log.gz 2024-09-16T13:20:23.688 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65280.log 2024-09-16T13:20:23.688 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58766.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58766.log.gz 2024-09-16T13:20:23.702 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-16T13:20:23.703 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65280.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65280.log.gz 2024-09-16T13:20:23.709 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45249.log 2024-09-16T13:20:23.726 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62685.log 2024-09-16T13:20:23.727 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45249.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45249.log.gz 2024-09-16T13:20:23.735 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70300.log 2024-09-16T13:20:23.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62685.log: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60250.log 2024-09-16T13:20:23.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70300.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70300.log.gz 2024-09-16T13:20:23.749 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62685.log.gz 2024-09-16T13:20:23.759 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50609.log 2024-09-16T13:20:23.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60250.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60250.log.gz 2024-09-16T13:20:23.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53677.log 2024-09-16T13:20:23.775 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50609.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50609.log.gz 2024-09-16T13:20:23.782 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56564.log 2024-09-16T13:20:23.791 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53677.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40520.log.gz 2024-09-16T13:20:23.791 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49506.log 2024-09-16T13:20:23.792 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56564.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56564.log.gz 2024-09-16T13:20:23.792 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55058.log 2024-09-16T13:20:23.793 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49506.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49506.log.gz 2024-09-16T13:20:23.794 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43371.log 2024-09-16T13:20:23.795 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55058.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55058.log.gz 2024-09-16T13:20:23.795 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66273.log 2024-09-16T13:20:23.796 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43371.log: 87.7% 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43371.log.gz 2024-09-16T13:20:23.796 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53677.log.gz 2024-09-16T13:20:23.796 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.44170.log 2024-09-16T13:20:23.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66273.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66273.log.gz 2024-09-16T13:20:23.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64165.log 2024-09-16T13:20:23.799 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.44170.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51683.log.gz 2024-09-16T13:20:23.800 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.44170.log.gz 2024-09-16T13:20:23.801 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51465.log 2024-09-16T13:20:23.801 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64165.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64165.log.gz 2024-09-16T13:20:23.802 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69130.log 2024-09-16T13:20:23.817 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75141.log 2024-09-16T13:20:23.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69130.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69130.log.gz 2024-09-16T13:20:23.824 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40736.log 2024-09-16T13:20:23.827 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75141.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75141.log.gz 2024-09-16T13:20:23.836 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72710.log 2024-09-16T13:20:23.837 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40736.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40736.log.gz 2024-09-16T13:20:23.848 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48170.log 2024-09-16T13:20:23.850 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72710.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72710.log.gz 2024-09-16T13:20:23.857 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60141.log 2024-09-16T13:20:23.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48170.log: 89.6% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46867.log.gz 2024-09-16T13:20:23.867 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48170.log.gz 2024-09-16T13:20:23.867 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74487.log 2024-09-16T13:20:23.867 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60141.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60141.log.gz 2024-09-16T13:20:23.868 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48517.log 2024-09-16T13:20:23.875 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74487.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39999.log.gz 2024-09-16T13:20:23.877 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69288.log 2024-09-16T13:20:23.879 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48517.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48517.log.gz 2024-09-16T13:20:23.879 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49038.log 2024-09-16T13:20:23.880 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69288.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69288.log.gz 2024-09-16T13:20:23.880 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56406.log 2024-09-16T13:20:23.881 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49038.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49038.log.gz 2024-09-16T13:20:23.881 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63345.log 2024-09-16T13:20:23.883 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56406.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56406.log.gz 2024-09-16T13:20:23.883 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71943.log 2024-09-16T13:20:23.884 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63345.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63345.log.gz 2024-09-16T13:20:23.884 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54596.log 2024-09-16T13:20:23.886 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71943.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71943.log.gz 2024-09-16T13:20:23.886 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57586.log 2024-09-16T13:20:23.887 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54596.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54596.log.gz 2024-09-16T13:20:23.887 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49909.log 2024-09-16T13:20:23.888 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57586.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57586.log.gz 2024-09-16T13:20:23.888 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73106.log 2024-09-16T13:20:23.899 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68368.log 2024-09-16T13:20:23.900 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73106.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73106.log.gz 2024-09-16T13:20:23.914 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49348.log 2024-09-16T13:20:23.914 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68368.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68368.log.gz 2024-09-16T13:20:23.915 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44631.log 2024-09-16T13:20:23.916 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49348.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49348.log.gz 2024-09-16T13:20:23.930 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60045.log 2024-09-16T13:20:23.936 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33080.log 2024-09-16T13:20:23.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41802.log 2024-09-16T13:20:23.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.33080.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60045.log.gz 2024-09-16T13:20:23.951 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61088.log 2024-09-16T13:20:23.951 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.33080.log.gz 2024-09-16T13:20:23.957 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41802.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69873.log 2024-09-16T13:20:23.965 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61088.log: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47141.log 2024-09-16T13:20:23.965 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61088.log.gz 2024-09-16T13:20:23.968 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69873.log: 93.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69873.log.gz 2024-09-16T13:20:23.968 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47668.log.gz 2024-09-16T13:20:23.968 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69580.log 2024-09-16T13:20:23.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47141.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47141.log.gz 2024-09-16T13:20:23.971 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68136.log 2024-09-16T13:20:23.971 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41802.log.gz 2024-09-16T13:20:23.971 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69580.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69580.log.gz 2024-09-16T13:20:23.971 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40020.log 2024-09-16T13:20:23.973 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68136.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68136.log.gz 2024-09-16T13:20:23.973 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64037.log 2024-09-16T13:20:23.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40020.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40020.log.gz 2024-09-16T13:20:23.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57883.log 2024-09-16T13:20:23.977 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64037.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64037.log.gz 2024-09-16T13:20:23.977 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53896.log 2024-09-16T13:20:23.978 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57883.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57883.log.gz 2024-09-16T13:20:23.978 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41909.log 2024-09-16T13:20:23.979 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53896.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53896.log.gz 2024-09-16T13:20:23.980 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66852.log 2024-09-16T13:20:23.990 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66748.log 2024-09-16T13:20:23.990 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66852.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66852.log.gz 2024-09-16T13:20:24.004 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67764.log 2024-09-16T13:20:24.007 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66748.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66748.log.gz 2024-09-16T13:20:24.007 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55275.log 2024-09-16T13:20:24.014 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67764.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65922.log 2024-09-16T13:20:24.014 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67764.log.gz 2024-09-16T13:20:24.027 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57345.log 2024-09-16T13:20:24.027 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55275.log.gz 2024-09-16T13:20:24.027 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65922.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65922.log.gz 2024-09-16T13:20:24.038 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49184.log.gz 2024-09-16T13:20:24.050 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46937.log 2024-09-16T13:20:24.050 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57345.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57345.log.gz 2024-09-16T13:20:24.059 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63566.log 2024-09-16T13:20:24.072 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46937.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46937.log.gz 2024-09-16T13:20:24.078 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72560.log 2024-09-16T13:20:24.079 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63566.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63566.log.gz 2024-09-16T13:20:24.087 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61900.log 2024-09-16T13:20:24.096 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72560.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72560.log.gz 2024-09-16T13:20:24.099 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48370.log.gz 2024-09-16T13:20:24.099 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75946.log 2024-09-16T13:20:24.102 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61900.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61900.log.gz 2024-09-16T13:20:24.103 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55106.log 2024-09-16T13:20:24.103 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75946.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75946.log.gz 2024-09-16T13:20:24.103 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67846.log 2024-09-16T13:20:24.103 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55106.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55106.log.gz 2024-09-16T13:20:24.104 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73026.log 2024-09-16T13:20:24.106 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67846.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67846.log.gz 2024-09-16T13:20:24.106 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56540.log 2024-09-16T13:20:24.106 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73026.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73026.log.gz 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57745.log 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56540.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56540.log.gz 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52989.log 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57745.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49909.log.gz 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.51225.log 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57745.log.gz 2024-09-16T13:20:24.109 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69243.log 2024-09-16T13:20:24.111 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.51225.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52989.log.gz 2024-09-16T13:20:24.111 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57192.log 2024-09-16T13:20:24.112 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69243.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69243.log.gz 2024-09-16T13:20:24.112 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67017.log 2024-09-16T13:20:24.113 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57192.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57192.log.gz 2024-09-16T13:20:24.113 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41341.log 2024-09-16T13:20:24.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67017.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67017.log.gz 2024-09-16T13:20:24.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50824.log 2024-09-16T13:20:24.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41341.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41341.log.gz 2024-09-16T13:20:24.116 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48890.log 2024-09-16T13:20:24.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67348.log 2024-09-16T13:20:24.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48890.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48890.log.gz 2024-09-16T13:20:24.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64731.log 2024-09-16T13:20:24.143 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67348.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67348.log.gz 2024-09-16T13:20:24.146 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64710.log 2024-09-16T13:20:24.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64731.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64731.log.gz 2024-09-16T13:20:24.165 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70188.log 2024-09-16T13:20:24.165 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64710.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64710.log.gz 2024-09-16T13:20:24.172 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66418.log 2024-09-16T13:20:24.173 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70188.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70188.log.gz 2024-09-16T13:20:24.175 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.51225.log.gz 2024-09-16T13:20:24.175 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70660.log 2024-09-16T13:20:24.177 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66418.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66418.log.gz 2024-09-16T13:20:24.177 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56468.log 2024-09-16T13:20:24.178 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70660.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70660.log.gz 2024-09-16T13:20:24.178 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50688.log 2024-09-16T13:20:24.179 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56468.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56468.log.gz 2024-09-16T13:20:24.179 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55568.log 2024-09-16T13:20:24.181 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50688.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50688.log.gz 2024-09-16T13:20:24.181 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36708.log 2024-09-16T13:20:24.183 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55568.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55568.log.gz 2024-09-16T13:20:24.183 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69355.log 2024-09-16T13:20:24.183 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36708.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36708.log.gz 2024-09-16T13:20:24.185 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46787.log 2024-09-16T13:20:24.185 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69355.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69355.log.gz 2024-09-16T13:20:24.185 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41757.log 2024-09-16T13:20:24.194 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38653.log 2024-09-16T13:20:24.195 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41757.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41757.log.gz 2024-09-16T13:20:24.209 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58979.log 2024-09-16T13:20:24.209 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38653.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.38653.log.gz 2024-09-16T13:20:24.210 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59900.log 2024-09-16T13:20:24.210 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58979.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58979.log.gz 2024-09-16T13:20:24.224 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46277.log 2024-09-16T13:20:24.224 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59900.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59900.log.gz 2024-09-16T13:20:24.231 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51762.log 2024-09-16T13:20:24.231 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46277.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46277.log.gz 2024-09-16T13:20:24.240 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68838.log 2024-09-16T13:20:24.245 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51762.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44631.log.gz 2024-09-16T13:20:24.250 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56295.log 2024-09-16T13:20:24.251 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68838.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51465.log.gz 2024-09-16T13:20:24.251 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68838.log.gz 2024-09-16T13:20:24.252 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60117.log 2024-09-16T13:20:24.252 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56295.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56295.log.gz 2024-09-16T13:20:24.252 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51720.log 2024-09-16T13:20:24.253 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60117.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60117.log.gz 2024-09-16T13:20:24.262 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72917.log 2024-09-16T13:20:24.264 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51720.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51720.log.gz 2024-09-16T13:20:24.264 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55299.log 2024-09-16T13:20:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72917.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72917.log.gz 2024-09-16T13:20:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65239.log 2024-09-16T13:20:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55299.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55299.log.gz 2024-09-16T13:20:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44705.log 2024-09-16T13:20:24.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65239.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65239.log.gz 2024-09-16T13:20:24.268 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55821.log 2024-09-16T13:20:24.270 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44705.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44705.log.gz 2024-09-16T13:20:24.270 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68053.log 2024-09-16T13:20:24.271 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55821.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55821.log.gz 2024-09-16T13:20:24.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44455.log 2024-09-16T13:20:24.272 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68053.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68053.log.gz 2024-09-16T13:20:24.272 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46765.log 2024-09-16T13:20:24.274 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44455.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44455.log.gz 2024-09-16T13:20:24.274 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65901.log 2024-09-16T13:20:24.275 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46765.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46765.log.gz 2024-09-16T13:20:24.275 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64414.log 2024-09-16T13:20:24.276 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65901.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65901.log.gz 2024-09-16T13:20:24.277 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65860.log 2024-09-16T13:20:24.278 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64414.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64414.log.gz 2024-09-16T13:20:24.278 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52461.log 2024-09-16T13:20:24.278 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65860.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65860.log.gz 2024-09-16T13:20:24.279 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58060.log 2024-09-16T13:20:24.282 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52461.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52461.log.gz 2024-09-16T13:20:24.282 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40498.log 2024-09-16T13:20:24.283 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58060.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58060.log.gz 2024-09-16T13:20:24.283 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69625.log 2024-09-16T13:20:24.284 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40498.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40498.log.gz 2024-09-16T13:20:24.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57952.log 2024-09-16T13:20:24.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69625.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69625.log.gz 2024-09-16T13:20:24.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46917.log 2024-09-16T13:20:24.288 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57952.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57952.log.gz 2024-09-16T13:20:24.289 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70233.log 2024-09-16T13:20:24.289 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46917.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46917.log.gz 2024-09-16T13:20:24.290 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37795.log 2024-09-16T13:20:24.292 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70233.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70233.log.gz 2024-09-16T13:20:24.292 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46617.log 2024-09-16T13:20:24.292 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37795.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37795.log.gz 2024-09-16T13:20:24.293 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74926.log 2024-09-16T13:20:24.294 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46617.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46617.log.gz 2024-09-16T13:20:24.294 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.47385.log 2024-09-16T13:20:24.306 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40760.log 2024-09-16T13:20:24.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.47385.log: 92.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65177.log 2024-09-16T13:20:24.320 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39148.log 2024-09-16T13:20:24.320 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.65177.log: -- replaced with /var/log/ceph/cluster1-client.mirror.40760.log.gz 2024-09-16T13:20:24.320 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65177.log.gz 2024-09-16T13:20:24.325 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.47385.log.gz 2024-09-16T13:20:24.338 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59921.log 2024-09-16T13:20:24.339 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.39148.log: 66.9% -- replaced with /var/log/ceph/cluster2-client.admin.39148.log.gz 2024-09-16T13:20:24.351 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69828.log 2024-09-16T13:20:24.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59921.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59921.log.gz 2024-09-16T13:20:24.366 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74487.log.gz 2024-09-16T13:20:24.366 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66335.log 2024-09-16T13:20:24.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69828.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69828.log.gz 2024-09-16T13:20:24.367 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37753.log 2024-09-16T13:20:24.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66335.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66335.log.gz 2024-09-16T13:20:24.370 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48687.log 2024-09-16T13:20:24.370 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37753.log.gz 2024-09-16T13:20:24.370 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38234.log 2024-09-16T13:20:24.370 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48687.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48687.log.gz 2024-09-16T13:20:24.370 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45827.log 2024-09-16T13:20:24.378 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38234.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46787.log.gz 2024-09-16T13:20:24.378 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73483.log 2024-09-16T13:20:24.384 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45827.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45827.log.gz 2024-09-16T13:20:24.385 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54255.log 2024-09-16T13:20:24.386 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73483.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73483.log.gz 2024-09-16T13:20:24.386 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69558.log 2024-09-16T13:20:24.387 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54255.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54255.log.gz 2024-09-16T13:20:24.387 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51128.log 2024-09-16T13:20:24.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69558.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69558.log.gz 2024-09-16T13:20:24.388 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68568.log 2024-09-16T13:20:24.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51128.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51128.log.gz 2024-09-16T13:20:24.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36997.log 2024-09-16T13:20:24.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68568.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68568.log.gz 2024-09-16T13:20:24.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62327.log 2024-09-16T13:20:24.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36997.log.gz 2024-09-16T13:20:24.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48870.log 2024-09-16T13:20:24.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62327.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62327.log.gz 2024-09-16T13:20:24.393 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45480.log 2024-09-16T13:20:24.394 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48870.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48870.log.gz 2024-09-16T13:20:24.394 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48724.log 2024-09-16T13:20:24.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45480.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45480.log.gz 2024-09-16T13:20:24.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61043.log 2024-09-16T13:20:24.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48724.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48724.log.gz 2024-09-16T13:20:24.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50284.log 2024-09-16T13:20:24.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61043.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61043.log.gz 2024-09-16T13:20:24.400 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73885.log 2024-09-16T13:20:24.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50284.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50284.log.gz 2024-09-16T13:20:24.404 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-16T13:20:24.405 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73885.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73885.log.gz 2024-09-16T13:20:24.406 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66128.log 2024-09-16T13:20:24.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50530.log 2024-09-16T13:20:24.425 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66128.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66128.log.gz 2024-09-16T13:20:24.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54789.log 2024-09-16T13:20:24.430 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50530.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50530.log.gz 2024-09-16T13:20:24.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48212.log 2024-09-16T13:20:24.440 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54789.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54789.log.gz 2024-09-16T13:20:24.458 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43840.log 2024-09-16T13:20:24.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71878.log 2024-09-16T13:20:24.477 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43840.log: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54351.log 2024-09-16T13:20:24.477 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71878.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71878.log.gz 2024-09-16T13:20:24.480 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43840.log.gz 2024-09-16T13:20:24.485 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67888.log 2024-09-16T13:20:24.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54351.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41909.log.gz 2024-09-16T13:20:24.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69423.log 2024-09-16T13:20:24.497 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67888.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67888.log.gz 2024-09-16T13:20:24.498 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54351.log.gz 2024-09-16T13:20:24.523 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65321.log 2024-09-16T13:20:24.524 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69423.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69423.log.gz 2024-09-16T13:20:24.538 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49142.log 2024-09-16T13:20:24.547 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41024.log 2024-09-16T13:20:24.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49142.log: 89.7% 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65321.log.gz 2024-09-16T13:20:24.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69333.log 2024-09-16T13:20:24.557 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41024.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74926.log.gz 2024-09-16T13:20:24.560 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49142.log.gz 2024-09-16T13:20:24.569 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74826.log 2024-09-16T13:20:24.570 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69333.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69333.log.gz 2024-09-16T13:20:24.580 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45784.log 2024-09-16T13:20:24.581 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74826.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74826.log.gz 2024-09-16T13:20:24.585 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50414.log 2024-09-16T13:20:24.590 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45784.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45784.log.gz 2024-09-16T13:20:24.590 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41024.log.gz 2024-09-16T13:20:24.591 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43980.log 2024-09-16T13:20:24.592 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50414.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50824.log.gz 2024-09-16T13:20:24.594 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67702.log 2024-09-16T13:20:24.594 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43980.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43980.log.gz 2024-09-16T13:20:24.594 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73751.log 2024-09-16T13:20:24.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67702.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67702.log.gz 2024-09-16T13:20:24.596 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41130.log 2024-09-16T13:20:24.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73751.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73751.log.gz 2024-09-16T13:20:24.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60294.log 2024-09-16T13:20:24.598 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41130.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41130.log.gz 2024-09-16T13:20:24.598 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44861.log 2024-09-16T13:20:24.599 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60294.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60294.log.gz 2024-09-16T13:20:24.599 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48500.log 2024-09-16T13:20:24.601 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44861.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44861.log.gz 2024-09-16T13:20:24.601 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60971.log 2024-09-16T13:20:24.602 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48500.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48500.log.gz 2024-09-16T13:20:24.602 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51187.log 2024-09-16T13:20:24.604 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60971.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60971.log.gz 2024-09-16T13:20:24.604 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52806.log 2024-09-16T13:20:24.611 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51187.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38234.log.gz 2024-09-16T13:20:24.611 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69783.log 2024-09-16T13:20:24.612 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52806.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52806.log.gz 2024-09-16T13:20:24.612 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37627.log 2024-09-16T13:20:24.612 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69783.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69783.log.gz 2024-09-16T13:20:24.615 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75231.log 2024-09-16T13:20:24.615 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37627.log.gz 2024-09-16T13:20:24.615 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61670.log 2024-09-16T13:20:24.615 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75231.log.gz 2024-09-16T13:20:24.615 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45698.log 2024-09-16T13:20:24.616 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61670.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61670.log.gz 2024-09-16T13:20:24.617 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61602.log 2024-09-16T13:20:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45698.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45698.log.gz 2024-09-16T13:20:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43074.log 2024-09-16T13:20:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61602.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61602.log.gz 2024-09-16T13:20:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70683.log 2024-09-16T13:20:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43074.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43074.log.gz 2024-09-16T13:20:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61168.log 2024-09-16T13:20:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70683.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70683.log.gz 2024-09-16T13:20:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67203.log 2024-09-16T13:20:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61168.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61168.log.gz 2024-09-16T13:20:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70724.log 2024-09-16T13:20:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67203.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67203.log.gz 2024-09-16T13:20:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69805.log 2024-09-16T13:20:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70724.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70724.log.gz 2024-09-16T13:20:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54620.log 2024-09-16T13:20:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69805.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69805.log.gz 2024-09-16T13:20:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47306.log 2024-09-16T13:20:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54620.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54620.log.gz 2024-09-16T13:20:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63470.log 2024-09-16T13:20:24.636 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47306.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47306.log.gz 2024-09-16T13:20:24.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68301.log 2024-09-16T13:20:24.637 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63470.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63470.log.gz 2024-09-16T13:20:24.637 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55592.log 2024-09-16T13:20:24.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68301.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68301.log.gz 2024-09-16T13:20:24.639 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67929.log 2024-09-16T13:20:24.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55592.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55592.log.gz 2024-09-16T13:20:24.640 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49966.log 2024-09-16T13:20:24.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67929.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67929.log.gz 2024-09-16T13:20:24.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40476.log 2024-09-16T13:20:24.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49966.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49966.log.gz 2024-09-16T13:20:24.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52368.log 2024-09-16T13:20:24.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40476.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40476.log.gz 2024-09-16T13:20:24.649 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61067.log 2024-09-16T13:20:24.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52368.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52368.log.gz 2024-09-16T13:20:24.651 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44086.log 2024-09-16T13:20:24.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61067.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61067.log.gz 2024-09-16T13:20:24.652 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46085.log 2024-09-16T13:20:24.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44086.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44086.log.gz 2024-09-16T13:20:24.654 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53242.log 2024-09-16T13:20:24.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46085.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46085.log.gz 2024-09-16T13:20:24.660 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65529.log 2024-09-16T13:20:24.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53242.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53242.log.gz 2024-09-16T13:20:24.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64507.log 2024-09-16T13:20:24.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65529.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65529.log.gz 2024-09-16T13:20:24.663 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39182.log 2024-09-16T13:20:24.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64507.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64507.log.gz 2024-09-16T13:20:24.666 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50104.log 2024-09-16T13:20:24.666 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39182.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39182.log.gz 2024-09-16T13:20:24.667 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53424.log 2024-09-16T13:20:24.667 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50104.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50104.log.gz 2024-09-16T13:20:24.668 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74867.log 2024-09-16T13:20:24.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53424.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53424.log.gz 2024-09-16T13:20:24.670 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41317.log 2024-09-16T13:20:24.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74867.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74867.log.gz 2024-09-16T13:20:24.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62021.log 2024-09-16T13:20:24.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41317.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41317.log.gz 2024-09-16T13:20:24.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39267.log 2024-09-16T13:20:24.674 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62021.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62021.log.gz 2024-09-16T13:20:24.674 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48704.log 2024-09-16T13:20:24.675 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39267.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39267.log.gz 2024-09-16T13:20:24.675 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56891.log 2024-09-16T13:20:24.675 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48704.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48704.log.gz 2024-09-16T13:20:24.678 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38789.log 2024-09-16T13:20:24.678 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56891.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56891.log.gz 2024-09-16T13:20:24.678 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46808.log 2024-09-16T13:20:24.678 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38789.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38789.log.gz 2024-09-16T13:20:24.680 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74050.log 2024-09-16T13:20:24.680 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46808.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46808.log.gz 2024-09-16T13:20:24.680 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-16T13:20:24.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74050.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74050.log.gz 2024-09-16T13:20:24.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47547.log 2024-09-16T13:20:24.703 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55655.log 2024-09-16T13:20:24.704 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47547.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47547.log.gz 2024-09-16T13:20:24.715 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44313.log 2024-09-16T13:20:24.716 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55655.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48212.log.gz 2024-09-16T13:20:24.716 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55655.log.gz 2024-09-16T13:20:24.716 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51762.log.gz 2024-09-16T13:20:24.717 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60204.log 2024-09-16T13:20:24.717 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50765.log 2024-09-16T13:20:24.718 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60204.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60204.log.gz 2024-09-16T13:20:24.718 INFO:teuthology.orchestra.run.smithi191.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44313.log.gz 2024-09-16T13:20:24.718 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64855.log 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.50765.log: -- /var/log/ceph/cluster2-client.mirror.74177.log 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50765.log.gz 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55034.log 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74177.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64855.log.gz 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74177.log.gz 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68156.log 2024-09-16T13:20:24.724 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.55034.log: /var/log/ceph/cluster1-client.mirror.65033.log 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55034.log.gz 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69108.log 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.65033.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65033.log.gz 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38177.log 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69108.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69108.log.gz 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59304.log 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5%/var/log/ceph/cluster1-client.mirror.1.38177.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68156.log.gz 2024-09-16T13:20:24.725 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62729.log 2024-09-16T13:20:24.727 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59304.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59304.log.gz 2024-09-16T13:20:24.727 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57658.log 2024-09-16T13:20:24.727 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62729.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62729.log.gz 2024-09-16T13:20:24.728 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74029.log 2024-09-16T13:20:24.728 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57658.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57658.log.gz 2024-09-16T13:20:24.730 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52143.log 2024-09-16T13:20:24.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74029.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74029.log.gz 2024-09-16T13:20:24.730 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45741.log 2024-09-16T13:20:24.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52143.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52143.log.gz 2024-09-16T13:20:24.731 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48746.log 2024-09-16T13:20:24.736 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45741.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45741.log.gz 2024-09-16T13:20:24.737 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67265.log 2024-09-16T13:20:24.756 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45271.log 2024-09-16T13:20:24.757 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67265.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67265.log.gz 2024-09-16T13:20:24.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70030.log 2024-09-16T13:20:24.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45271.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38177.log.gz 2024-09-16T13:20:24.770 INFO:teuthology.orchestra.run.smithi191.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45271.log.gz 2024-09-16T13:20:24.770 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66232.log 2024-09-16T13:20:24.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70030.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70030.log.gz 2024-09-16T13:20:24.771 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58931.log 2024-09-16T13:20:24.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66232.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66232.log.gz 2024-09-16T13:20:24.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54181.log 2024-09-16T13:20:24.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58931.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58931.log.gz 2024-09-16T13:20:24.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49162.log 2024-09-16T13:20:24.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54181.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54181.log.gz 2024-09-16T13:20:24.776 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63542.log 2024-09-16T13:20:24.776 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49162.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49162.log.gz 2024-09-16T13:20:24.776 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.44742.log 2024-09-16T13:20:24.776 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63542.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63542.log.gz 2024-09-16T13:20:24.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43884.log 2024-09-16T13:20:24.795 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.44742.log: 92.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72961.log 2024-09-16T13:20:24.795 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43884.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43884.log.gz 2024-09-16T13:20:24.795 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.44742.log.gz 2024-09-16T13:20:24.795 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50572.log 2024-09-16T13:20:24.797 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72961.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72961.log.gz 2024-09-16T13:20:24.797 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.56688.log 2024-09-16T13:20:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48249.log 2024-09-16T13:20:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.56688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56153.log 2024-09-16T13:20:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48249.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48249.log.gz 2024-09-16T13:20:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52503.log 2024-09-16T13:20:24.839 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56153.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56988.log 2024-09-16T13:20:24.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52503.log: 89.8% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56153.log.gz 2024-09-16T13:20:24.848 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.56688.log.gz 2024-09-16T13:20:24.848 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70120.log 2024-09-16T13:20:24.849 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56988.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56988.log.gz 2024-09-16T13:20:24.852 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52503.log.gz 2024-09-16T13:20:24.866 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75053.log 2024-09-16T13:20:24.866 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70120.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70120.log.gz 2024-09-16T13:20:24.869 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50414.log.gz 2024-09-16T13:20:24.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43096.log 2024-09-16T13:20:24.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75053.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75053.log.gz 2024-09-16T13:20:24.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50903.log 2024-09-16T13:20:24.883 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49647.log 2024-09-16T13:20:24.892 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62181.log 2024-09-16T13:20:24.892 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49647.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49647.log.gz 2024-09-16T13:20:24.901 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64438.log 2024-09-16T13:20:24.914 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62181.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62181.log.gz 2024-09-16T13:20:24.921 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61236.log 2024-09-16T13:20:24.922 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64438.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64438.log.gz 2024-09-16T13:20:24.930 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75356.log 2024-09-16T13:20:24.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66500.log 2024-09-16T13:20:24.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75356.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75356.log.gz 2024-09-16T13:20:24.944 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61236.log.gz 2024-09-16T13:20:24.957 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74740.log 2024-09-16T13:20:24.960 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66500.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66500.log.gz 2024-09-16T13:20:24.966 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67970.log 2024-09-16T13:20:24.976 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47626.log 2024-09-16T13:20:24.980 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67970.log: 90.5% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74740.log.gz 2024-09-16T13:20:24.984 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42513.log 2024-09-16T13:20:24.990 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47626.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67970.log.gz 2024-09-16T13:20:24.995 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50572.log.gz 2024-09-16T13:20:24.998 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47626.log.gz 2024-09-16T13:20:25.004 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61260.log 2024-09-16T13:20:25.005 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42513.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42513.log.gz 2024-09-16T13:20:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47326.log 2024-09-16T13:20:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61260.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61260.log.gz 2024-09-16T13:20:25.021 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51187.log.gz 2024-09-16T13:20:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54661.log 2024-09-16T13:20:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47326.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47326.log.gz 2024-09-16T13:20:25.034 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38091.log 2024-09-16T13:20:25.036 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54661.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54661.log.gz 2024-09-16T13:20:25.036 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49125.log 2024-09-16T13:20:25.036 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38091.log: 81.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38091.log.gz 2024-09-16T13:20:25.038 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55410.log 2024-09-16T13:20:25.038 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49125.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49125.log.gz 2024-09-16T13:20:25.038 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65342.log 2024-09-16T13:20:25.040 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55410.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55410.log.gz 2024-09-16T13:20:25.040 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67784.log 2024-09-16T13:20:25.041 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65342.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65342.log.gz 2024-09-16T13:20:25.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41641.log 2024-09-16T13:20:25.044 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67784.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67784.log.gz 2024-09-16T13:20:25.044 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75903.log 2024-09-16T13:20:25.046 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41641.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41641.log.gz 2024-09-16T13:20:25.047 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47609.log 2024-09-16T13:20:25.047 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75903.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75903.log.gz 2024-09-16T13:20:25.048 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71899.log 2024-09-16T13:20:25.048 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47609.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47609.log.gz 2024-09-16T13:20:25.049 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66769.log 2024-09-16T13:20:25.050 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71899.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71899.log.gz 2024-09-16T13:20:25.050 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52967.log 2024-09-16T13:20:25.051 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66769.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66769.log.gz 2024-09-16T13:20:25.052 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47183.log 2024-09-16T13:20:25.053 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52967.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52967.log.gz 2024-09-16T13:20:25.054 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45678.log 2024-09-16T13:20:25.065 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61844.log 2024-09-16T13:20:25.071 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45678.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37921.log 2024-09-16T13:20:25.075 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61844.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45678.log.gz 2024-09-16T13:20:25.084 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64282.log 2024-09-16T13:20:25.085 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37921.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37921.log.gz 2024-09-16T13:20:25.090 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48746.log.gz 2024-09-16T13:20:25.090 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63386.log 2024-09-16T13:20:25.092 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64282.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64282.log.gz 2024-09-16T13:20:25.092 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45443.log 2024-09-16T13:20:25.095 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63386.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63386.log.gz 2024-09-16T13:20:25.095 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46638.log 2024-09-16T13:20:25.100 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45443.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45443.log.gz 2024-09-16T13:20:25.101 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50204.log 2024-09-16T13:20:25.102 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46638.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46638.log.gz 2024-09-16T13:20:25.102 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68973.log 2024-09-16T13:20:25.103 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50204.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50204.log.gz 2024-09-16T13:20:25.103 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56843.log 2024-09-16T13:20:25.105 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68973.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68973.log.gz 2024-09-16T13:20:25.105 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62351.log 2024-09-16T13:20:25.106 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56843.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56843.log.gz 2024-09-16T13:20:25.106 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53740.log 2024-09-16T13:20:25.107 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62351.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62351.log.gz 2024-09-16T13:20:25.107 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63450.log 2024-09-16T13:20:25.109 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53740.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53740.log.gz 2024-09-16T13:20:25.109 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40890.log 2024-09-16T13:20:25.110 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63450.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63450.log.gz 2024-09-16T13:20:25.111 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55347.log 2024-09-16T13:20:25.112 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40890.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40890.log.gz 2024-09-16T13:20:25.112 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48111.log 2024-09-16T13:20:25.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55347.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55347.log.gz 2024-09-16T13:20:25.114 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73728.log 2024-09-16T13:20:25.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48111.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48111.log.gz 2024-09-16T13:20:25.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63085.log 2024-09-16T13:20:25.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73728.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73728.log.gz 2024-09-16T13:20:25.116 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63366.log 2024-09-16T13:20:25.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63085.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63085.log.gz 2024-09-16T13:20:25.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53039.log 2024-09-16T13:20:25.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63366.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63366.log.gz 2024-09-16T13:20:25.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66108.log 2024-09-16T13:20:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53039.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53039.log.gz 2024-09-16T13:20:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41845.log 2024-09-16T13:20:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66108.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66108.log.gz 2024-09-16T13:20:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55994.log 2024-09-16T13:20:25.129 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41845.log: 93.3% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41845.log.gz 2024-09-16T13:20:25.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69513.log 2024-09-16T13:20:25.130 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55994.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55994.log.gz 2024-09-16T13:20:25.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64552.log 2024-09-16T13:20:25.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69513.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69513.log.gz 2024-09-16T13:20:25.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58472.log 2024-09-16T13:20:25.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64552.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61844.log.gz 2024-09-16T13:20:25.132 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64552.log.gz 2024-09-16T13:20:25.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51703.log 2024-09-16T13:20:25.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58472.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58472.log.gz 2024-09-16T13:20:25.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69378.log 2024-09-16T13:20:25.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51703.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51703.log.gz 2024-09-16T13:20:25.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49808.log 2024-09-16T13:20:25.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69378.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69378.log.gz 2024-09-16T13:20:25.137 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63408.log 2024-09-16T13:20:25.138 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49808.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49808.log.gz 2024-09-16T13:20:25.138 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66459.log 2024-09-16T13:20:25.139 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63408.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63408.log.gz 2024-09-16T13:20:25.140 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47830.log 2024-09-16T13:20:25.140 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66459.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66459.log.gz 2024-09-16T13:20:25.141 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46728.log 2024-09-16T13:20:25.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47830.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47830.log.gz 2024-09-16T13:20:25.142 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52722.log 2024-09-16T13:20:25.142 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46728.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46728.log.gz 2024-09-16T13:20:25.143 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71920.log 2024-09-16T13:20:25.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52722.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52722.log.gz 2024-09-16T13:20:25.145 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41495.log 2024-09-16T13:20:25.146 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71920.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71920.log.gz 2024-09-16T13:20:25.146 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65301.log 2024-09-16T13:20:25.147 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41495.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41495.log.gz 2024-09-16T13:20:25.147 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44679.log 2024-09-16T13:20:25.148 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65301.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65301.log.gz 2024-09-16T13:20:25.148 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47982.log 2024-09-16T13:20:25.149 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.44679.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44679.log.gz 2024-09-16T13:20:25.149 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70098.log 2024-09-16T13:20:25.150 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47982.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47982.log.gz 2024-09-16T13:20:25.151 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57417.log 2024-09-16T13:20:25.151 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70098.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70098.log.gz 2024-09-16T13:20:25.151 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59433.log 2024-09-16T13:20:25.152 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57417.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57417.log.gz 2024-09-16T13:20:25.152 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47289.log 2024-09-16T13:20:25.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59433.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59433.log.gz 2024-09-16T13:20:25.153 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63273.log 2024-09-16T13:20:25.154 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47289.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47289.log.gz 2024-09-16T13:20:25.154 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61108.log 2024-09-16T13:20:25.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63273.log.gz 2024-09-16T13:20:25.156 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70570.log 2024-09-16T13:20:25.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61108.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61108.log.gz 2024-09-16T13:20:25.158 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60584.log 2024-09-16T13:20:25.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70570.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70570.log.gz 2024-09-16T13:20:25.159 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67722.log 2024-09-16T13:20:25.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60584.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60584.log.gz 2024-09-16T13:20:25.160 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72044.log 2024-09-16T13:20:25.162 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67722.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67722.log.gz 2024-09-16T13:20:25.163 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49447.log 2024-09-16T13:20:25.164 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72044.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72044.log.gz 2024-09-16T13:20:25.165 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62139.log 2024-09-16T13:20:25.165 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49447.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49447.log.gz 2024-09-16T13:20:25.165 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74236.log 2024-09-16T13:20:25.167 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62139.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62139.log.gz 2024-09-16T13:20:25.167 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67826.log 2024-09-16T13:20:25.187 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52854.log 2024-09-16T13:20:25.188 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67826.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67826.log.gz 2024-09-16T13:20:25.206 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64370.log 2024-09-16T13:20:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52854.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52854.log.gz 2024-09-16T13:20:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62593.log 2024-09-16T13:20:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64370.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64370.log.gz 2024-09-16T13:20:25.244 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.67612.log 2024-09-16T13:20:25.245 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62593.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62593.log.gz 2024-09-16T13:20:25.257 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60753.log 2024-09-16T13:20:25.264 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.67612.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.67612.log.gz 2024-09-16T13:20:25.267 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67389.log 2024-09-16T13:20:25.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60753.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60753.log.gz 2024-09-16T13:20:25.270 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47183.log.gz 2024-09-16T13:20:25.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40387.log 2024-09-16T13:20:25.273 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67389.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67389.log.gz 2024-09-16T13:20:25.273 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50493.log 2024-09-16T13:20:25.273 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40387.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40387.log.gz 2024-09-16T13:20:25.274 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58911.log 2024-09-16T13:20:25.282 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50493.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50903.log.gz 2024-09-16T13:20:25.282 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62094.log 2024-09-16T13:20:25.284 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58911.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58911.log.gz 2024-09-16T13:20:25.284 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65446.log 2024-09-16T13:20:25.285 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62094.log.gz 2024-09-16T13:20:25.285 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42752.log 2024-09-16T13:20:25.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65446.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65446.log.gz 2024-09-16T13:20:25.288 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67058.log 2024-09-16T13:20:25.289 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42752.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42752.log.gz 2024-09-16T13:20:25.289 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40435.log 2024-09-16T13:20:25.292 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67058.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67058.log.gz 2024-09-16T13:20:25.292 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72245.log 2024-09-16T13:20:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40435.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40435.log.gz 2024-09-16T13:20:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50283.log 2024-09-16T13:20:25.295 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72245.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72245.log.gz 2024-09-16T13:20:25.295 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42725.log 2024-09-16T13:20:25.297 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50283.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50283.log.gz 2024-09-16T13:20:25.297 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46703.log 2024-09-16T13:20:25.298 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42725.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42725.log.gz 2024-09-16T13:20:25.298 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46317.log 2024-09-16T13:20:25.299 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46703.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46703.log.gz 2024-09-16T13:20:25.299 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57462.log 2024-09-16T13:20:25.301 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46317.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46317.log.gz 2024-09-16T13:20:25.301 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68748.log 2024-09-16T13:20:25.303 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57462.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57462.log.gz 2024-09-16T13:20:25.303 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43488.log 2024-09-16T13:20:25.305 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68748.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68748.log.gz 2024-09-16T13:20:25.305 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65197.log 2024-09-16T13:20:25.307 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43488.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43096.log.gz 2024-09-16T13:20:25.308 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69603.log 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65197.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65197.log.gz 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43488.log.gz 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45400.log 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46477.log 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69603.log.gz 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75988.log 2024-09-16T13:20:25.312 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46477.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46477.log.gz 2024-09-16T13:20:25.314 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.58581.log 2024-09-16T13:20:25.314 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75988.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75988.log.gz 2024-09-16T13:20:25.314 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59413.log 2024-09-16T13:20:25.317 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.58581.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45400.log.gz 2024-09-16T13:20:25.317 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44021.log 2024-09-16T13:20:25.318 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59413.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59413.log.gz 2024-09-16T13:20:25.319 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63518.log 2024-09-16T13:20:25.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44021.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44021.log.gz 2024-09-16T13:20:25.325 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40064.log 2024-09-16T13:20:25.325 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63518.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63518.log.gz 2024-09-16T13:20:25.326 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45807.log 2024-09-16T13:20:25.326 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40064.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40064.log.gz 2024-09-16T13:20:25.327 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51165.log 2024-09-16T13:20:25.332 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45807.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45807.log.gz 2024-09-16T13:20:25.333 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68500.log 2024-09-16T13:20:25.333 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51165.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51165.log.gz 2024-09-16T13:20:25.334 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55725.log 2024-09-16T13:20:25.334 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68500.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68500.log.gz 2024-09-16T13:20:25.335 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42920.log 2024-09-16T13:20:25.335 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55725.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55725.log.gz 2024-09-16T13:20:25.336 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45936.log 2024-09-16T13:20:25.336 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42920.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42920.log.gz 2024-09-16T13:20:25.337 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56223.log 2024-09-16T13:20:25.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45936.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45936.log.gz 2024-09-16T13:20:25.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75693.log 2024-09-16T13:20:25.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56223.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56223.log.gz 2024-09-16T13:20:25.344 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64324.log 2024-09-16T13:20:25.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.75693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75693.log.gz 2024-09-16T13:20:25.345 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66872.log 2024-09-16T13:20:25.346 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64324.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64324.log.gz 2024-09-16T13:20:25.346 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45065.log 2024-09-16T13:20:25.348 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66872.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66872.log.gz 2024-09-16T13:20:25.349 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41249.log 2024-09-16T13:20:25.354 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45065.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45065.log.gz 2024-09-16T13:20:25.355 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63123.log 2024-09-16T13:20:25.356 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41249.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41249.log.gz 2024-09-16T13:20:25.356 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67306.log 2024-09-16T13:20:25.358 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63123.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63123.log.gz 2024-09-16T13:20:25.358 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52062.log 2024-09-16T13:20:25.360 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67306.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67306.log.gz 2024-09-16T13:20:25.361 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52250.log 2024-09-16T13:20:25.361 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52062.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52062.log.gz 2024-09-16T13:20:25.361 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55950.log 2024-09-16T13:20:25.362 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52250.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52250.log.gz 2024-09-16T13:20:25.363 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38755.log 2024-09-16T13:20:25.364 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55950.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55950.log.gz 2024-09-16T13:20:25.364 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56018.log 2024-09-16T13:20:25.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38755.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38755.log.gz 2024-09-16T13:20:25.365 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37669.log 2024-09-16T13:20:25.366 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56018.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56018.log.gz 2024-09-16T13:20:25.366 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61284.log 2024-09-16T13:20:25.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37669.log.gz 2024-09-16T13:20:25.367 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62801.log 2024-09-16T13:20:25.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61284.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61284.log.gz 2024-09-16T13:20:25.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44479.log 2024-09-16T13:20:25.369 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62801.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62801.log.gz 2024-09-16T13:20:25.369 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72748.log 2024-09-16T13:20:25.370 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44479.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44479.log.gz 2024-09-16T13:20:25.370 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42989.log 2024-09-16T13:20:25.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72748.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72748.log.gz 2024-09-16T13:20:25.372 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74194.log 2024-09-16T13:20:25.377 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42989.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42989.log.gz 2024-09-16T13:20:25.378 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55010.log 2024-09-16T13:20:25.379 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74194.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74194.log.gz 2024-09-16T13:20:25.379 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74345.log 2024-09-16T13:20:25.380 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55010.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55010.log.gz 2024-09-16T13:20:25.380 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45870.log 2024-09-16T13:20:25.381 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74345.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74345.log.gz 2024-09-16T13:20:25.382 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47929.log 2024-09-16T13:20:25.389 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45870.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45870.log.gz 2024-09-16T13:20:25.389 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54416.log 2024-09-16T13:20:25.389 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47929.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47929.log.gz 2024-09-16T13:20:25.389 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54548.log 2024-09-16T13:20:25.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54416.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54416.log.gz 2024-09-16T13:20:25.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38346.log 2024-09-16T13:20:25.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54548.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54548.log.gz 2024-09-16T13:20:25.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50392.log 2024-09-16T13:20:25.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38346.log: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38346.log.gz 2024-09-16T13:20:25.393 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55705.log 2024-09-16T13:20:25.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50392.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50392.log.gz 2024-09-16T13:20:25.394 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48019.log 2024-09-16T13:20:25.395 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55705.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55705.log.gz 2024-09-16T13:20:25.395 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68658.log 2024-09-16T13:20:25.396 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48019.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48019.log.gz 2024-09-16T13:20:25.396 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70742.log 2024-09-16T13:20:25.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68658.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74236.log.gz 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68658.log.gz 2024-09-16T13:20:25.397 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:25.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53605.log 2024-09-16T13:20:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37837.log 2024-09-16T13:20:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70742.log: /var/log/ceph/cluster1-client.mirror.53605.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70742.log.gz 2024-09-16T13:20:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53653.log 2024-09-16T13:20:25.399 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53605.log.gz 2024-09-16T13:20:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37837.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37837.log.gz 2024-09-16T13:20:25.400 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67368.log 2024-09-16T13:20:25.400 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41430.log 2024-09-16T13:20:25.400 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53653.log: /var/log/ceph/cluster2-client.mirror.67368.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53653.log.gz 2024-09-16T13:20:25.401 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66087.log 2024-09-16T13:20:25.403 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41430.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67368.log.gz 2024-09-16T13:20:25.403 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60411.log 2024-09-16T13:20:25.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66087.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66087.log.gz 2024-09-16T13:20:25.404 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48190.log 2024-09-16T13:20:25.407 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60411.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60411.log.gz 2024-09-16T13:20:25.407 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66397.log 2024-09-16T13:20:25.407 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48190.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48190.log.gz 2024-09-16T13:20:25.407 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41430.log.gz 2024-09-16T13:20:25.409 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50651.log 2024-09-16T13:20:25.409 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.66397.log: -- /var/log/ceph/cluster1-client.mirror.41781.log 2024-09-16T13:20:25.409 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.50651.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66397.log.gz 2024-09-16T13:20:25.409 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43170.log 2024-09-16T13:20:25.409 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41781.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41781.log.gz 2024-09-16T13:20:25.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45377.log 2024-09-16T13:20:25.411 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43170.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43170.log.gz 2024-09-16T13:20:25.412 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51562.log 2024-09-16T13:20:25.417 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45377.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45377.log.gz 2024-09-16T13:20:25.418 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65550.log 2024-09-16T13:20:25.418 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51562.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51562.log.gz 2024-09-16T13:20:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56588.log 2024-09-16T13:20:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65550.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65550.log.gz 2024-09-16T13:20:25.421 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74465.log 2024-09-16T13:20:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56588.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56588.log.gz 2024-09-16T13:20:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74094.log 2024-09-16T13:20:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74465.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74465.log.gz 2024-09-16T13:20:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58831.log 2024-09-16T13:20:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74094.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74094.log.gz 2024-09-16T13:20:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73663.log 2024-09-16T13:20:25.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58831.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58831.log.gz 2024-09-16T13:20:25.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69040.log 2024-09-16T13:20:25.428 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73663.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73663.log.gz 2024-09-16T13:20:25.428 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.74963.log 2024-09-16T13:20:25.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69040.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69040.log.gz 2024-09-16T13:20:25.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46397.log 2024-09-16T13:20:25.432 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.74963.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.74963.log.gz 2024-09-16T13:20:25.432 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48232.log 2024-09-16T13:20:25.433 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46397.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46397.log.gz 2024-09-16T13:20:25.434 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68523.log 2024-09-16T13:20:25.434 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48232.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48232.log.gz 2024-09-16T13:20:25.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51661.log 2024-09-16T13:20:25.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68523.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68523.log.gz 2024-09-16T13:20:25.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37205.log 2024-09-16T13:20:25.437 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51661.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51661.log.gz 2024-09-16T13:20:25.437 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58452.log 2024-09-16T13:20:25.437 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37205.log.gz 2024-09-16T13:20:25.438 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51314.log 2024-09-16T13:20:25.440 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58452.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58452.log.gz 2024-09-16T13:20:25.441 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70413.log 2024-09-16T13:20:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51314.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51314.log.gz 2024-09-16T13:20:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52416.log 2024-09-16T13:20:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70413.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70413.log.gz 2024-09-16T13:20:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50008.log 2024-09-16T13:20:25.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52416.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52416.log.gz 2024-09-16T13:20:25.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42448.log 2024-09-16T13:20:25.445 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50008.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50008.log.gz 2024-09-16T13:20:25.445 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40594.log 2024-09-16T13:20:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42448.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42448.log.gz 2024-09-16T13:20:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73705.log 2024-09-16T13:20:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40594.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40594.log.gz 2024-09-16T13:20:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38687.log 2024-09-16T13:20:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73705.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73705.log.gz 2024-09-16T13:20:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59003.log 2024-09-16T13:20:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38687.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38687.log.gz 2024-09-16T13:20:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37585.log 2024-09-16T13:20:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59003.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59003.log.gz 2024-09-16T13:20:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69085.log 2024-09-16T13:20:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37585.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37585.log.gz 2024-09-16T13:20:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57931.log 2024-09-16T13:20:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69085.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69085.log.gz 2024-09-16T13:20:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44544.log 2024-09-16T13:20:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57931.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57931.log.gz 2024-09-16T13:20:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59457.log 2024-09-16T13:20:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44544.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44544.log.gz 2024-09-16T13:20:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.61844.log 2024-09-16T13:20:25.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59457.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59457.log.gz 2024-09-16T13:20:25.469 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67430.log 2024-09-16T13:20:25.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.61844.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.58581.log.gz 2024-09-16T13:20:25.470 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47567.log 2024-09-16T13:20:25.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67430.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67430.log.gz 2024-09-16T13:20:25.473 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62974.log 2024-09-16T13:20:25.474 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47567.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47567.log.gz 2024-09-16T13:20:25.474 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64057.log 2024-09-16T13:20:25.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-09-16T13:20:25.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62974.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62974.log.gz 2024-09-16T13:20:25.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38832.log 2024-09-16T13:20:25.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64057.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64057.log.gz 2024-09-16T13:20:25.478 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52702.log 2024-09-16T13:20:25.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.38832.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38832.log.gz 2024-09-16T13:20:25.479 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52481.log 2024-09-16T13:20:25.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52702.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52702.log.gz 2024-09-16T13:20:25.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53083.log 2024-09-16T13:20:25.484 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52481.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52481.log.gz 2024-09-16T13:20:25.484 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66914.log 2024-09-16T13:20:25.485 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53083.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53083.log.gz 2024-09-16T13:20:25.485 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65818.log 2024-09-16T13:20:25.486 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66914.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66914.log.gz 2024-09-16T13:20:25.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41024.log 2024-09-16T13:20:25.489 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65818.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65818.log.gz 2024-09-16T13:20:25.489 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72940.log 2024-09-16T13:20:25.505 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.41024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44953.log 2024-09-16T13:20:25.506 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72940.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72940.log.gz 2024-09-16T13:20:25.519 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51096.log 2024-09-16T13:20:25.519 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44953.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44953.log.gz 2024-09-16T13:20:25.524 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.61844.log.gz 2024-09-16T13:20:25.524 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67141.log 2024-09-16T13:20:25.538 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43908.log 2024-09-16T13:20:25.538 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67141.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67141.log.gz 2024-09-16T13:20:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55147.log 2024-09-16T13:20:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43908.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43908.log.gz 2024-09-16T13:20:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73505.log 2024-09-16T13:20:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55147.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55147.log.gz 2024-09-16T13:20:25.555 INFO:teuthology.orchestra.run.smithi191.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41024.log.gz 2024-09-16T13:20:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66170.log 2024-09-16T13:20:25.573 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.73505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56688.log 2024-09-16T13:20:25.574 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66170.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66170.log.gz 2024-09-16T13:20:25.584 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42848.log 2024-09-16T13:20:25.593 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56688.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56688.log.gz 2024-09-16T13:20:25.593 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50493.log.gz 2024-09-16T13:20:25.594 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56042.log 2024-09-16T13:20:25.595 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42848.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42848.log.gz 2024-09-16T13:20:25.595 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44742.log 2024-09-16T13:20:25.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56042.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56042.log.gz 2024-09-16T13:20:25.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40219.log 2024-09-16T13:20:25.606 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44742.log: 94.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74761.log 2024-09-16T13:20:25.607 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40219.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44742.log.gz 2024-09-16T13:20:25.607 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.40219.log.gz 2024-09-16T13:20:25.608 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69490.log 2024-09-16T13:20:25.608 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74761.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74761.log.gz 2024-09-16T13:20:25.610 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47810.log 2024-09-16T13:20:25.610 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69490.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69490.log.gz 2024-09-16T13:20:25.610 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66480.log 2024-09-16T13:20:25.612 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47810.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47810.log.gz 2024-09-16T13:20:25.612 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.48411.log 2024-09-16T13:20:25.613 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66480.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66480.log.gz 2024-09-16T13:20:25.613 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68703.log 2024-09-16T13:20:25.625 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.48411.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41822.log 2024-09-16T13:20:25.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68703.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68703.log.gz 2024-09-16T13:20:25.627 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.48411.log.gz 2024-09-16T13:20:25.627 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69310.log 2024-09-16T13:20:25.633 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41822.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41822.log.gz 2024-09-16T13:20:25.634 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65122.log 2024-09-16T13:20:25.634 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69310.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69310.log.gz 2024-09-16T13:20:25.635 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56271.log 2024-09-16T13:20:25.637 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.65122.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65122.log.gz 2024-09-16T13:20:25.637 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56364.log 2024-09-16T13:20:25.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56271.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56271.log.gz 2024-09-16T13:20:25.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47161.log 2024-09-16T13:20:25.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56364.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56364.log.gz 2024-09-16T13:20:25.641 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47011.log 2024-09-16T13:20:25.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47161.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47161.log.gz 2024-09-16T13:20:25.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40295.log 2024-09-16T13:20:25.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47011.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47011.log.gz 2024-09-16T13:20:25.644 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47951.log 2024-09-16T13:20:25.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40295.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40295.log.gz 2024-09-16T13:20:25.645 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69715.log 2024-09-16T13:20:25.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59345.log 2024-09-16T13:20:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69715.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69715.log.gz 2024-09-16T13:20:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58358.log 2024-09-16T13:20:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59345.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59345.log.gz 2024-09-16T13:20:25.689 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.45480.log 2024-09-16T13:20:25.689 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58358.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58358.log.gz 2024-09-16T13:20:25.701 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68590.log 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.45480.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51096.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51582.log 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68590.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68590.log.gz 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51582.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51582.log.gz 2024-09-16T13:20:25.715 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47209.log 2024-09-16T13:20:25.720 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2.log: 86.4% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-16T13:20:25.720 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44587.log 2024-09-16T13:20:25.720 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47209.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47209.log.gz 2024-09-16T13:20:25.742 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41389.log 2024-09-16T13:20:25.747 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44587.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44587.log.gz 2024-09-16T13:20:25.754 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.45480.log.gz 2024-09-16T13:20:25.755 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68434.log 2024-09-16T13:20:25.757 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41389.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41389.log.gz 2024-09-16T13:20:25.757 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44652.log 2024-09-16T13:20:25.758 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68434.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68434.log.gz 2024-09-16T13:20:25.758 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44170.log 2024-09-16T13:20:25.760 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.44652.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44652.log.gz 2024-09-16T13:20:25.760 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50124.log 2024-09-16T13:20:25.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44170.log: 80.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44170.log.gz 2024-09-16T13:20:25.761 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57504.log 2024-09-16T13:20:25.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50124.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50124.log.gz 2024-09-16T13:20:25.762 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59574.log 2024-09-16T13:20:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57504.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57504.log.gz 2024-09-16T13:20:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43443.log 2024-09-16T13:20:25.766 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59574.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59574.log.gz 2024-09-16T13:20:25.766 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38350.log 2024-09-16T13:20:25.768 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43443.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43443.log.gz 2024-09-16T13:20:25.768 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54837.log 2024-09-16T13:20:25.768 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38350.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38350.log.gz 2024-09-16T13:20:25.769 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63229.log 2024-09-16T13:20:25.770 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54837.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54837.log.gz 2024-09-16T13:20:25.770 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70480.log 2024-09-16T13:20:25.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63229.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63229.log.gz 2024-09-16T13:20:25.771 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52099.log 2024-09-16T13:20:25.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70480.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70480.log.gz 2024-09-16T13:20:25.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55544.log 2024-09-16T13:20:25.774 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52099.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52099.log.gz 2024-09-16T13:20:25.774 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53304.log 2024-09-16T13:20:25.775 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55544.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55544.log.gz 2024-09-16T13:20:25.775 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28139.log 2024-09-16T13:20:25.777 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53304.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53304.log.gz 2024-09-16T13:20:25.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67410.log 2024-09-16T13:20:25.777 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.tmp-client.admin.28139.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28139.log.gz 2024-09-16T13:20:25.778 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75281.log 2024-09-16T13:20:25.778 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.67410.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67410.log.gz 2024-09-16T13:20:25.780 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47246.log 2024-09-16T13:20:25.780 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.75281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75281.log.gz 2024-09-16T13:20:25.780 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37477.log 2024-09-16T13:20:25.782 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47246.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47246.log.gz 2024-09-16T13:20:25.782 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58431.log 2024-09-16T13:20:25.782 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37477.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37477.log.gz 2024-09-16T13:20:25.784 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54902.log 2024-09-16T13:20:25.784 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58431.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58431.log.gz 2024-09-16T13:20:25.784 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63063.log 2024-09-16T13:20:25.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54902.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54902.log.gz 2024-09-16T13:20:25.786 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48411.log 2024-09-16T13:20:25.789 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63063.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63063.log.gz 2024-09-16T13:20:25.789 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.38234.log 2024-09-16T13:20:25.791 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.48411.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48411.log.gz 2024-09-16T13:20:25.791 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40868.log 2024-09-16T13:20:25.799 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.3.38234.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50651.log.gz 2024-09-16T13:20:25.799 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51947.log 2024-09-16T13:20:25.805 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40868.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40868.log.gz 2024-09-16T13:20:25.806 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48803.log 2024-09-16T13:20:25.817 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58718.log 2024-09-16T13:20:25.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48803.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48803.log.gz 2024-09-16T13:20:25.831 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38704.log 2024-09-16T13:20:25.831 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58718.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58718.log.gz 2024-09-16T13:20:25.842 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72064.log 2024-09-16T13:20:25.842 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38704.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38704.log.gz 2024-09-16T13:20:25.851 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66707.log 2024-09-16T13:20:25.851 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72064.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72064.log.gz 2024-09-16T13:20:25.856 INFO:teuthology.orchestra.run.smithi191.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-16T13:20:25.858 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50782.log 2024-09-16T13:20:25.858 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66707.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66707.log.gz 2024-09-16T13:20:25.859 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46377.log 2024-09-16T13:20:25.860 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50782.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50782.log.gz 2024-09-16T13:20:25.861 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61582.log 2024-09-16T13:20:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46377.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46377.log.gz 2024-09-16T13:20:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45291.log 2024-09-16T13:20:25.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61582.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61582.log.gz 2024-09-16T13:20:25.864 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48783.log 2024-09-16T13:20:25.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45291.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45291.log.gz 2024-09-16T13:20:25.871 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48912.log 2024-09-16T13:20:25.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48783.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48783.log.gz 2024-09-16T13:20:25.872 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52392.log 2024-09-16T13:20:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54924.log 2024-09-16T13:20:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52392.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52392.log.gz 2024-09-16T13:20:25.901 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74006.log 2024-09-16T13:20:25.902 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54924.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54924.log.gz 2024-09-16T13:20:25.902 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62635.log 2024-09-16T13:20:25.913 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.74006.log: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68177.log 2024-09-16T13:20:25.913 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74006.log.gz 2024-09-16T13:20:25.914 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62635.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62635.log.gz 2024-09-16T13:20:25.926 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74156.log 2024-09-16T13:20:25.926 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68177.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68177.log.gz 2024-09-16T13:20:25.938 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59072.log 2024-09-16T13:20:25.940 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.74156.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74156.log.gz 2024-09-16T13:20:25.947 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40825.log 2024-09-16T13:20:25.948 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59072.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59072.log.gz 2024-09-16T13:20:25.963 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58669.log 2024-09-16T13:20:25.968 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40825.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40825.log.gz 2024-09-16T13:20:25.976 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52320.log 2024-09-16T13:20:25.977 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58669.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58669.log.gz 2024-09-16T13:20:25.985 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63022.log 2024-09-16T13:20:25.995 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46171.log 2024-09-16T13:20:25.998 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63022.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52320.log.gz 2024-09-16T13:20:26.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46597.log 2024-09-16T13:20:26.007 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46171.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63022.log.gz 2024-09-16T13:20:26.012 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62998.log 2024-09-16T13:20:26.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46597.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73505.log.gz 2024-09-16T13:20:26.019 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61955.log 2024-09-16T13:20:26.019 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62998.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62998.log.gz 2024-09-16T13:20:26.021 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59828.log 2024-09-16T13:20:26.021 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61955.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46171.log.gz 2024-09-16T13:20:26.021 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61955.log.gz 2024-09-16T13:20:26.021 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60366.log 2024-09-16T13:20:26.023 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59828.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59828.log.gz 2024-09-16T13:20:26.023 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70075.log 2024-09-16T13:20:26.024 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60366.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60366.log.gz 2024-09-16T13:20:26.024 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52344.log 2024-09-16T13:20:26.025 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70075.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70075.log.gz 2024-09-16T13:20:26.025 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62613.log 2024-09-16T13:20:26.026 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52344.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46597.log.gz 2024-09-16T13:20:26.027 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52344.log.gz 2024-09-16T13:20:26.027 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54118.log 2024-09-16T13:20:26.029 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62613.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62613.log.gz 2024-09-16T13:20:26.029 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50940.log 2024-09-16T13:20:26.030 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54118.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54118.log.gz 2024-09-16T13:20:26.031 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55472.log 2024-09-16T13:20:26.033 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50940.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.38234.log.gz 2024-09-16T13:20:26.033 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37077.log 2024-09-16T13:20:26.034 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55472.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55472.log.gz 2024-09-16T13:20:26.036 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72437.log 2024-09-16T13:20:26.036 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37077.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37077.log.gz 2024-09-16T13:20:26.036 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60093.log 2024-09-16T13:20:26.037 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72437.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72437.log.gz 2024-09-16T13:20:26.037 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69265.log 2024-09-16T13:20:26.038 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60093.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60093.log.gz 2024-09-16T13:20:26.039 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50940.log.gz 2024-09-16T13:20:26.039 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43207.log 2024-09-16T13:20:26.039 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69265.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69265.log.gz 2024-09-16T13:20:26.040 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64189.log 2024-09-16T13:20:26.041 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.43207.log: -- /var/log/ceph/cluster1-client.mirror.73192.log 2024-09-16T13:20:26.041 INFO:teuthology.orchestra.run.smithi191.stderr: 74.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43207.log.gz 2024-09-16T13:20:26.041 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64527.log 2024-09-16T13:20:26.041 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64189.log.gz 2024-09-16T13:20:26.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.73192.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73192.log.gz 2024-09-16T13:20:26.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70638.log 2024-09-16T13:20:26.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.64527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47124.log 2024-09-16T13:20:26.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70638.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70638.log.gz 2024-09-16T13:20:26.043 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38297.log 2024-09-16T13:20:26.044 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47124.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47124.log.gz 2024-09-16T13:20:26.044 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64527.log.gz 2024-09-16T13:20:26.044 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54861.log 2024-09-16T13:20:26.046 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.38297.log: -- /var/log/ceph/cluster1-client.mirror.69985.log 2024-09-16T13:20:26.046 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54861.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38297.log.gz 2024-09-16T13:20:26.047 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54861.log.gz 2024-09-16T13:20:26.047 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63587.log 2024-09-16T13:20:26.047 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53200.log 2024-09-16T13:20:26.047 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69985.log.gz 2024-09-16T13:20:26.047 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75097.log 2024-09-16T13:20:26.048 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53200.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63587.log.gz 2024-09-16T13:20:26.048 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53200.log.gz 2024-09-16T13:20:26.049 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46437.log 2024-09-16T13:20:26.049 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.75097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75377.log 2024-09-16T13:20:26.050 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46437.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75097.log.gz 2024-09-16T13:20:26.052 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54159.log 2024-09-16T13:20:26.052 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2%/var/log/ceph/cluster1-client.admin.75377.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46437.log.gz 2024-09-16T13:20:26.052 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75377.log.gz 2024-09-16T13:20:26.052 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38346.log 2024-09-16T13:20:26.052 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.54159.log: -- /var/log/ceph/cluster2-client.mirror.58309.log 2024-09-16T13:20:26.054 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.38346.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54159.log.gz 2024-09-16T13:20:26.054 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44929.log 2024-09-16T13:20:26.054 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58309.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58309.log.gz 2024-09-16T13:20:26.054 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72005.log 2024-09-16T13:20:26.056 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44929.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44929.log.gz 2024-09-16T13:20:26.056 INFO:teuthology.orchestra.run.smithi191.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38346.log.gz 2024-09-16T13:20:26.056 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40339.log 2024-09-16T13:20:26.056 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46194.log 2024-09-16T13:20:26.056 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40339.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40339.log.gz 2024-09-16T13:20:26.057 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70053.log 2024-09-16T13:20:26.058 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46194.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72005.log.gz 2024-09-16T13:20:26.059 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43932.log 2024-09-16T13:20:26.059 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70053.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70053.log.gz 2024-09-16T13:20:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52926.log 2024-09-16T13:20:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43932.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43932.log.gz 2024-09-16T13:20:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61489.log 2024-09-16T13:20:26.063 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52926.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52926.log.gz 2024-09-16T13:20:26.063 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46194.log.gz 2024-09-16T13:20:26.063 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54279.log 2024-09-16T13:20:26.063 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61489.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61489.log.gz 2024-09-16T13:20:26.068 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66376.log 2024-09-16T13:20:26.068 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.54279.log: -- /var/log/ceph/cluster2-client.admin.38005.log 2024-09-16T13:20:26.068 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54279.log.gz 2024-09-16T13:20:26.068 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37417.log 2024-09-16T13:20:26.068 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.38005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38005.log.gz 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65777.log 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37417.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37417.log.gz 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66376.log.gz 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster1-client.mirror.59594.log 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60182.log 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65777.log: /var/log/ceph/cluster1-client.mirror.59594.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65777.log.gz 2024-09-16T13:20:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41517.log 2024-09-16T13:20:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60182.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59594.log.gz 2024-09-16T13:20:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46042.log 2024-09-16T13:20:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41517.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60182.log.gz 2024-09-16T13:20:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69918.log 2024-09-16T13:20:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46042.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46042.log.gz 2024-09-16T13:20:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66004.log 2024-09-16T13:20:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69918.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69918.log.gz 2024-09-16T13:20:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66438.log 2024-09-16T13:20:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66004.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66004.log.gz 2024-09-16T13:20:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46497.log 2024-09-16T13:20:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66438.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66438.log.gz 2024-09-16T13:20:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37291.log 2024-09-16T13:20:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46497.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46497.log.gz 2024-09-16T13:20:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57096.log 2024-09-16T13:20:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37291.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37291.log.gz 2024-09-16T13:20:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47268.log 2024-09-16T13:20:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57096.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57096.log.gz 2024-09-16T13:20:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57791.log 2024-09-16T13:20:26.097 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56799.log 2024-09-16T13:20:26.097 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57791.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47951.log.gz 2024-09-16T13:20:26.100 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57791.log.gz 2024-09-16T13:20:26.100 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66624.log 2024-09-16T13:20:26.100 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56799.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56799.log.gz 2024-09-16T13:20:26.101 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58811.log 2024-09-16T13:20:26.103 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66624.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66624.log.gz 2024-09-16T13:20:26.104 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66934.log 2024-09-16T13:20:26.106 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58811.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58811.log.gz 2024-09-16T13:20:26.106 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70435.log 2024-09-16T13:20:26.110 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66934.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66934.log.gz 2024-09-16T13:20:26.110 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58180.log 2024-09-16T13:20:26.110 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70435.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70435.log.gz 2024-09-16T13:20:26.110 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64141.log 2024-09-16T13:20:26.112 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58180.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58180.log.gz 2024-09-16T13:20:26.112 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70278.log 2024-09-16T13:20:26.113 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64141.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64141.log.gz 2024-09-16T13:20:26.113 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49585.log 2024-09-16T13:20:26.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70278.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70278.log.gz 2024-09-16T13:20:26.114 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51925.log 2024-09-16T13:20:26.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49585.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49585.log.gz 2024-09-16T13:20:26.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46357.log 2024-09-16T13:20:26.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51925.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51925.log.gz 2024-09-16T13:20:26.117 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40987.log 2024-09-16T13:20:26.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46357.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46357.log.gz 2024-09-16T13:20:26.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54500.log 2024-09-16T13:20:26.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40987.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40987.log.gz 2024-09-16T13:20:26.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54500.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54500.log.gz 2024-09-16T13:20:26.135 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51947.log.gz 2024-09-16T13:20:26.282 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48912.log.gz 2024-09-16T13:20:26.396 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47268.log.gz 2024-09-16T13:20:26.436 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41517.log.gz 2024-09-16T13:20:28.398 INFO:teuthology.orchestra.run.smithi177.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-16T13:20:28.471 INFO:teuthology.orchestra.run.smithi177.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-16T13:20:29.883 INFO:teuthology.orchestra.run.smithi177.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-16T13:20:29.885 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T13:20:29.885 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m8.328s 2024-09-16T13:20:29.885 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m22.810s 2024-09-16T13:20:29.885 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.859s 2024-09-16T13:21:08.516 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-16T13:21:10.128 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-16T13:21:23.126 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-16T13:21:23.128 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-16T13:21:23.129 INFO:teuthology.orchestra.run.smithi191.stderr:real 1m1.541s 2024-09-16T13:21:23.129 INFO:teuthology.orchestra.run.smithi191.stderr:user 2m51.848s 2024-09-16T13:21:23.129 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m7.614s 2024-09-16T13:21:23.129 INFO:tasks.ceph:Archiving logs... 2024-09-16T13:21:23.129 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi177/log 2024-09-16T13:21:23.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:21:26.246 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi191/log 2024-09-16T13:21:26.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:21:51.281 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T13:21:51.296 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T13:21:51.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:21:51.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:21:51.382 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T13:21:51.394 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T13:21:51.394 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:21:51.397 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:21:51.411 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:21:51.413 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv01.front.sepia.ceph.com 3 7 377 51 -346us[ -352us] +/- 49ms 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 +455us[ +448us] +/- 62ms 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 55 +70us[ +64us] +/- 71ms 2024-09-16T13:21:51.416 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 -400us[ -402us] +/- 49ms 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 +446us[ +445us] +/- 62ms 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +22us[ +21us] +/- 71ms 2024-09-16T13:21:51.418 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:21:51.419 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T13:21:51.431 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T13:21:51.431 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T13:21:51.443 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T13:21:51.525 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T13:21:51.593 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:21:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:21:51.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:21:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:21:51.652 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T13:21:51.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T13:21:51.682 INFO:teuthology.task.internal:Duration was 2738.788248 seconds 2024-09-16T13:21:51.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T13:21:51.695 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T13:21:51.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:21:51.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:21:51.736 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:21:51.738 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:21:51.995 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T13:21:51.996 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T13:21:51.996 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:21:52.063 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-09-16T13:21:52.064 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:21:52.084 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T13:21:52.084 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:21:52.114 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:21:52.156 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T13:21:52.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:21:52.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:21:52.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T13:21:52.709 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T13:21:52.710 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:21:52.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:21:52.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T13:21:52.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:21:52.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:21:52.880 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-09-16T13:21:52.882 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-09-16T13:21:52.899 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:21:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:21:52.954 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:21:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:21:52.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T13:21:52.980 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T13:21:52.981 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi177 2024-09-16T13:21:52.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:21:53.028 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907268/remote/smithi191 2024-09-16T13:21:53.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:21:53.061 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T13:21:53.062 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:21:53.066 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:21:53.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T13:21:53.126 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T13:21:53.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T13:21:53.137 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T13:21:53.137 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:21:53.140 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:21:53.152 INFO:teuthology.orchestra.run.smithi177.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:21 /home/ubuntu/cephtest 2024-09-16T13:21:53.166 INFO:teuthology.orchestra.run.smithi191.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:21 /home/ubuntu/cephtest 2024-09-16T13:21:53.167 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T13:21:53.180 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T13:21:53.212 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 2738.7882475852966 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-16T13:21:53.213 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T13:21:53.315 INFO:teuthology.run:pass